You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kie.apache.org by "Pere Fernandez (apache)" <pe...@apache.org> on 2024/09/30 20:32:57 UTC

[HEADS UP] KIE Flyway Add-On

Hi,

I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
to solve some issues we found when enabling Flyway to initialize the
application database in a compact setup (having runtime with other modules
that require persistence in the same app like data-index, jobs-service and
data-audit). In this setup, a single Flyway instance is executed for the
whole application during the startup and we are configuring it to apply all
the SQL scripts (sorted by the version specified in the file name) from all
those modules and keeping a single index for all the scripts applied during
the Flyway execution.

The main problems we identified are related to the versioning of those SQL
files inside the modules. Flyway gets script information (version,
description...) from the file name (Ej:
V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
version to be unique, it doesn't discriminate script source, description.
Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
V1.45.0.4...), in this scenario it is very likely to have collisions if
different modules use the same version in the script name.
Additionally, it's also possible that for the same reason, DB upgrades
won't work. For example if a module is updated to bring new SQL Scripts to
upgrade the DB (let's say data-index bringing V1.5.0) the same application
running a Flyway again won't be able to apply those changes if in a
previous run another add-on brought a script with a higher version (in this
case runtime-persistence has v10).

Another issue is that, in this setup, Flyway must be configured to know the
locations of all the SQL scripts in the applications.properties. In
general, this SQL files are in different locations, and there are modules
that provide SQL's for different DB types (h2, pgsql). This can be a bit
cumbersome to have it properly configured, specially for customers that are
not familiar with the internals of our code.


So, in order to help here, we are adding this Kie Flyway Add-On.

The idea is that instead of using the default Flyway integration provided
by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
running a separate instance of Flyway for each module that requires it and
using an independent index for each one them, this way we can ensure that
there will be no friction between SQL scripts. To make an module available
for KIE Flyway, first we must add the following dependency in our module
pom.xml:

<dependency>
  <groupId>org.kie</groupId>
  <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
</dependency>

And make sure we include a kie-flyway.properties descriptor located at `
src/main/resources/META-INF` with the following properties:

- module.name: logic name to identify the module.
- module.locations.<db_type>: this property map is used to setup the script
locations per each database supported by our module, (ej:
module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
possible to use the module.locations.default key if we want to use some
fallback SQL's in case the application Database type isn't supported. This
locations should refer to a path (or comma-separate paths) unique to the
module to avoid possible collisions and we should as much as possible using
default Flyway locations like db/migrations. You'll notice that I moved the
current SQL scripts from its location to
src/main/resources/kie-flyway/db/<module-name>/<db-type>

You can see an example of this file here (kie-flyway.properties
<https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties>)
and the new module locations here (locations
<https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc>
).


This addon supports the following configurations in the
application.properties file:

   - kie.flyway.enabled=true|false # Enables the execution of the KIE
   Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
   and fully disabled in Spring-boot.
   - kie.flyway.<module-name>.enabled=true|false # This is used to disable
   specific module if we need to


Some considerations:

   - This Add-On is intended to be used for development / test / examples
   and not recommended to use in production
   - As all our Add-Ons will use the default application datasource, so
   it's not possible to make it use a different
   - It doesn't cover all the Flyway features, it's only used to run
   migrations to initialize our components.
   - It doesn't replace the Platform Flyway in the kogito applications, the
   user can still use Flyway to load its own scripts without having to care of
   configuring anything from our modules.


At this point, for simplicity I only tried to address the DB Initialization
of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
user-tasks), but this feature can be also brought to the Data-Index & Jobs
Service if we want to, the only reason I didn't do it is to avoid doing
changes that could break the Sonataflow Operator and the examples, but I'll
happily apply the changes there if there's an agreement on this.

I think that's all... sorry for this long email but I think this
improvement required some explanations.

Thank you!

Pere

RE: [HEADS UP] KIE Flyway Add-On

Posted by Martin Weiler <mw...@ibm.com.INVALID>.
+1

Great work, Pere!

________________________________________
From: Tiago Bento <ti...@apache.org>
Sent: Monday, September 30, 2024 3:16 PM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [HEADS UP] KIE Flyway Add-On

Thanks for breaking it down so nicely Pere. Great improvement!

On Mon, Sep 30, 2024 at 4:47 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> Thank you very much for the detailed information, Pere!
>
> As you pointed out, this solved already identified issues when running
> the compact architecture setup.
>
> From my perspective, this is a big +1!
>
> On Mon, Sep 30, 2024 at 4:33 PM Pere Fernandez (apache)
> <pe...@apache.org> wrote:
> >
> > Hi,
> >
> > I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
> > to solve some issues we found when enabling Flyway to initialize the
> > application database in a compact setup (having runtime with other modules
> > that require persistence in the same app like data-index, jobs-service and
> > data-audit). In this setup, a single Flyway instance is executed for the
> > whole application during the startup and we are configuring it to apply all
> > the SQL scripts (sorted by the version specified in the file name) from all
> > those modules and keeping a single index for all the scripts applied during
> > the Flyway execution.
> >
> > The main problems we identified are related to the versioning of those SQL
> > files inside the modules. Flyway gets script information (version,
> > description...) from the file name (Ej:
> > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> > version to be unique, it doesn't discriminate script source, description.
> > Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > different modules use the same version in the script name.
> > Additionally, it's also possible that for the same reason, DB upgrades
> > won't work. For example if a module is updated to bring new SQL Scripts to
> > upgrade the DB (let's say data-index bringing V1.5.0) the same application
> > running a Flyway again won't be able to apply those changes if in a
> > previous run another add-on brought a script with a higher version (in this
> > case runtime-persistence has v10).
> >
> > Another issue is that, in this setup, Flyway must be configured to know the
> > locations of all the SQL scripts in the applications.properties. In
> > general, this SQL files are in different locations, and there are modules
> > that provide SQL's for different DB types (h2, pgsql). This can be a bit
> > cumbersome to have it properly configured, specially for customers that are
> > not familiar with the internals of our code.
> >
> >
> > So, in order to help here, we are adding this Kie Flyway Add-On.
> >
> > The idea is that instead of using the default Flyway integration provided
> > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > running a separate instance of Flyway for each module that requires it and
> > using an independent index for each one them, this way we can ensure that
> > there will be no friction between SQL scripts. To make an module available
> > for KIE Flyway, first we must add the following dependency in our module
> > pom.xml:
> >
> > <dependency>
> >   <groupId>org.kie</groupId>
> >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > </dependency>
> >
> > And make sure we include a kie-flyway.properties descriptor located at `
> > src/main/resources/META-INF` with the following properties:
> >
> > - module.name: logic name to identify the module.
> > - module.locations.<db_type>: this property map is used to setup the script
> > locations per each database supported by our module, (ej:
> > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > possible to use the module.locations.default key if we want to use some
> > fallback SQL's in case the application Database type isn't supported. This
> > locations should refer to a path (or comma-separate paths) unique to the
> > module to avoid possible collisions and we should as much as possible using
> > default Flyway locations like db/migrations. You'll notice that I moved the
> > current SQL scripts from its location to
> > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> >
> > You can see an example of this file here (kie-flyway.properties
> > <https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties >)
> > and the new module locations here (locations
> > <https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc >
> > ).
> >
> >
> > This addon supports the following configurations in the
> > application.properties file:
> >
> >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> >    Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
> >    and fully disabled in Spring-boot.
> >    - kie.flyway.<module-name>.enabled=true|false # This is used to disable
> >    specific module if we need to
> >
> >
> > Some considerations:
> >
> >    - This Add-On is intended to be used for development / test / examples
> >    and not recommended to use in production
> >    - As all our Add-Ons will use the default application datasource, so
> >    it's not possible to make it use a different
> >    - It doesn't cover all the Flyway features, it's only used to run
> >    migrations to initialize our components.
> >    - It doesn't replace the Platform Flyway in the kogito applications, the
> >    user can still use Flyway to load its own scripts without having to care of
> >    configuring anything from our modules.
> >
> >
> > At this point, for simplicity I only tried to address the DB Initialization
> > of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
> > user-tasks), but this feature can be also brought to the Data-Index & Jobs
> > Service if we want to, the only reason I didn't do it is to avoid doing
> > changes that could break the Sonataflow Operator and the examples, but I'll
> > happily apply the changes there if there's an agreement on this.
> >
> > I think that's all... sorry for this long email but I think this
> > improvement required some explanations.
> >
> > Thank you!
> >
> > Pere
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Tiago Bento <ti...@apache.org>.
Thanks for breaking it down so nicely Pere. Great improvement!

On Mon, Sep 30, 2024 at 4:47 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> Thank you very much for the detailed information, Pere!
>
> As you pointed out, this solved already identified issues when running
> the compact architecture setup.
>
> From my perspective, this is a big +1!
>
> On Mon, Sep 30, 2024 at 4:33 PM Pere Fernandez (apache)
> <pe...@apache.org> wrote:
> >
> > Hi,
> >
> > I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
> > to solve some issues we found when enabling Flyway to initialize the
> > application database in a compact setup (having runtime with other modules
> > that require persistence in the same app like data-index, jobs-service and
> > data-audit). In this setup, a single Flyway instance is executed for the
> > whole application during the startup and we are configuring it to apply all
> > the SQL scripts (sorted by the version specified in the file name) from all
> > those modules and keeping a single index for all the scripts applied during
> > the Flyway execution.
> >
> > The main problems we identified are related to the versioning of those SQL
> > files inside the modules. Flyway gets script information (version,
> > description...) from the file name (Ej:
> > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> > version to be unique, it doesn't discriminate script source, description.
> > Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > different modules use the same version in the script name.
> > Additionally, it's also possible that for the same reason, DB upgrades
> > won't work. For example if a module is updated to bring new SQL Scripts to
> > upgrade the DB (let's say data-index bringing V1.5.0) the same application
> > running a Flyway again won't be able to apply those changes if in a
> > previous run another add-on brought a script with a higher version (in this
> > case runtime-persistence has v10).
> >
> > Another issue is that, in this setup, Flyway must be configured to know the
> > locations of all the SQL scripts in the applications.properties. In
> > general, this SQL files are in different locations, and there are modules
> > that provide SQL's for different DB types (h2, pgsql). This can be a bit
> > cumbersome to have it properly configured, specially for customers that are
> > not familiar with the internals of our code.
> >
> >
> > So, in order to help here, we are adding this Kie Flyway Add-On.
> >
> > The idea is that instead of using the default Flyway integration provided
> > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > running a separate instance of Flyway for each module that requires it and
> > using an independent index for each one them, this way we can ensure that
> > there will be no friction between SQL scripts. To make an module available
> > for KIE Flyway, first we must add the following dependency in our module
> > pom.xml:
> >
> > <dependency>
> >   <groupId>org.kie</groupId>
> >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > </dependency>
> >
> > And make sure we include a kie-flyway.properties descriptor located at `
> > src/main/resources/META-INF` with the following properties:
> >
> > - module.name: logic name to identify the module.
> > - module.locations.<db_type>: this property map is used to setup the script
> > locations per each database supported by our module, (ej:
> > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > possible to use the module.locations.default key if we want to use some
> > fallback SQL's in case the application Database type isn't supported. This
> > locations should refer to a path (or comma-separate paths) unique to the
> > module to avoid possible collisions and we should as much as possible using
> > default Flyway locations like db/migrations. You'll notice that I moved the
> > current SQL scripts from its location to
> > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> >
> > You can see an example of this file here (kie-flyway.properties
> > <https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties>)
> > and the new module locations here (locations
> > <https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc>
> > ).
> >
> >
> > This addon supports the following configurations in the
> > application.properties file:
> >
> >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> >    Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
> >    and fully disabled in Spring-boot.
> >    - kie.flyway.<module-name>.enabled=true|false # This is used to disable
> >    specific module if we need to
> >
> >
> > Some considerations:
> >
> >    - This Add-On is intended to be used for development / test / examples
> >    and not recommended to use in production
> >    - As all our Add-Ons will use the default application datasource, so
> >    it's not possible to make it use a different
> >    - It doesn't cover all the Flyway features, it's only used to run
> >    migrations to initialize our components.
> >    - It doesn't replace the Platform Flyway in the kogito applications, the
> >    user can still use Flyway to load its own scripts without having to care of
> >    configuring anything from our modules.
> >
> >
> > At this point, for simplicity I only tried to address the DB Initialization
> > of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
> > user-tasks), but this feature can be also brought to the Data-Index & Jobs
> > Service if we want to, the only reason I didn't do it is to avoid doing
> > changes that could break the Sonataflow Operator and the examples, but I'll
> > happily apply the changes there if there's an agreement on this.
> >
> > I think that's all... sorry for this long email but I think this
> > improvement required some explanations.
> >
> > Thank you!
> >
> > Pere
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Alex Porcelli <al...@porcelli.me>.
Thank you very much for the detailed information, Pere!

As you pointed out, this solved already identified issues when running
the compact architecture setup.

From my perspective, this is a big +1!

On Mon, Sep 30, 2024 at 4:33 PM Pere Fernandez (apache)
<pe...@apache.org> wrote:
>
> Hi,
>
> I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
> to solve some issues we found when enabling Flyway to initialize the
> application database in a compact setup (having runtime with other modules
> that require persistence in the same app like data-index, jobs-service and
> data-audit). In this setup, a single Flyway instance is executed for the
> whole application during the startup and we are configuring it to apply all
> the SQL scripts (sorted by the version specified in the file name) from all
> those modules and keeping a single index for all the scripts applied during
> the Flyway execution.
>
> The main problems we identified are related to the versioning of those SQL
> files inside the modules. Flyway gets script information (version,
> description...) from the file name (Ej:
> V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> version to be unique, it doesn't discriminate script source, description.
> Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> V1.45.0.4...), in this scenario it is very likely to have collisions if
> different modules use the same version in the script name.
> Additionally, it's also possible that for the same reason, DB upgrades
> won't work. For example if a module is updated to bring new SQL Scripts to
> upgrade the DB (let's say data-index bringing V1.5.0) the same application
> running a Flyway again won't be able to apply those changes if in a
> previous run another add-on brought a script with a higher version (in this
> case runtime-persistence has v10).
>
> Another issue is that, in this setup, Flyway must be configured to know the
> locations of all the SQL scripts in the applications.properties. In
> general, this SQL files are in different locations, and there are modules
> that provide SQL's for different DB types (h2, pgsql). This can be a bit
> cumbersome to have it properly configured, specially for customers that are
> not familiar with the internals of our code.
>
>
> So, in order to help here, we are adding this Kie Flyway Add-On.
>
> The idea is that instead of using the default Flyway integration provided
> by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> running a separate instance of Flyway for each module that requires it and
> using an independent index for each one them, this way we can ensure that
> there will be no friction between SQL scripts. To make an module available
> for KIE Flyway, first we must add the following dependency in our module
> pom.xml:
>
> <dependency>
>   <groupId>org.kie</groupId>
>   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> </dependency>
>
> And make sure we include a kie-flyway.properties descriptor located at `
> src/main/resources/META-INF` with the following properties:
>
> - module.name: logic name to identify the module.
> - module.locations.<db_type>: this property map is used to setup the script
> locations per each database supported by our module, (ej:
> module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> possible to use the module.locations.default key if we want to use some
> fallback SQL's in case the application Database type isn't supported. This
> locations should refer to a path (or comma-separate paths) unique to the
> module to avoid possible collisions and we should as much as possible using
> default Flyway locations like db/migrations. You'll notice that I moved the
> current SQL scripts from its location to
> src/main/resources/kie-flyway/db/<module-name>/<db-type>
>
> You can see an example of this file here (kie-flyway.properties
> <https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties>)
> and the new module locations here (locations
> <https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc>
> ).
>
>
> This addon supports the following configurations in the
> application.properties file:
>
>    - kie.flyway.enabled=true|false # Enables the execution of the KIE
>    Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
>    and fully disabled in Spring-boot.
>    - kie.flyway.<module-name>.enabled=true|false # This is used to disable
>    specific module if we need to
>
>
> Some considerations:
>
>    - This Add-On is intended to be used for development / test / examples
>    and not recommended to use in production
>    - As all our Add-Ons will use the default application datasource, so
>    it's not possible to make it use a different
>    - It doesn't cover all the Flyway features, it's only used to run
>    migrations to initialize our components.
>    - It doesn't replace the Platform Flyway in the kogito applications, the
>    user can still use Flyway to load its own scripts without having to care of
>    configuring anything from our modules.
>
>
> At this point, for simplicity I only tried to address the DB Initialization
> of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
> user-tasks), but this feature can be also brought to the Data-Index & Jobs
> Service if we want to, the only reason I didn't do it is to avoid doing
> changes that could break the Sonataflow Operator and the examples, but I'll
> happily apply the changes there if there's an agreement on this.
>
> I think that's all... sorry for this long email but I think this
> improvement required some explanations.
>
> Thank you!
>
> Pere

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Daniel J dos Santos <da...@apache.org>.
Great work, Pere!

+1

On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> Hi,
> 
> I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
> to solve some issues we found when enabling Flyway to initialize the
> application database in a compact setup (having runtime with other modules
> that require persistence in the same app like data-index, jobs-service and
> data-audit). In this setup, a single Flyway instance is executed for the
> whole application during the startup and we are configuring it to apply all
> the SQL scripts (sorted by the version specified in the file name) from all
> those modules and keeping a single index for all the scripts applied during
> the Flyway execution.
> 
> The main problems we identified are related to the versioning of those SQL
> files inside the modules. Flyway gets script information (version,
> description...) from the file name (Ej:
> V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> version to be unique, it doesn't discriminate script source, description.
> Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> V1.45.0.4...), in this scenario it is very likely to have collisions if
> different modules use the same version in the script name.
> Additionally, it's also possible that for the same reason, DB upgrades
> won't work. For example if a module is updated to bring new SQL Scripts to
> upgrade the DB (let's say data-index bringing V1.5.0) the same application
> running a Flyway again won't be able to apply those changes if in a
> previous run another add-on brought a script with a higher version (in this
> case runtime-persistence has v10).
> 
> Another issue is that, in this setup, Flyway must be configured to know the
> locations of all the SQL scripts in the applications.properties. In
> general, this SQL files are in different locations, and there are modules
> that provide SQL's for different DB types (h2, pgsql). This can be a bit
> cumbersome to have it properly configured, specially for customers that are
> not familiar with the internals of our code.
> 
> 
> So, in order to help here, we are adding this Kie Flyway Add-On.
> 
> The idea is that instead of using the default Flyway integration provided
> by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> running a separate instance of Flyway for each module that requires it and
> using an independent index for each one them, this way we can ensure that
> there will be no friction between SQL scripts. To make an module available
> for KIE Flyway, first we must add the following dependency in our module
> pom.xml:
> 
> <dependency>
>   <groupId>org.kie</groupId>
>   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> </dependency>
> 
> And make sure we include a kie-flyway.properties descriptor located at `
> src/main/resources/META-INF` with the following properties:
> 
> - module.name: logic name to identify the module.
> - module.locations.<db_type>: this property map is used to setup the script
> locations per each database supported by our module, (ej:
> module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> possible to use the module.locations.default key if we want to use some
> fallback SQL's in case the application Database type isn't supported. This
> locations should refer to a path (or comma-separate paths) unique to the
> module to avoid possible collisions and we should as much as possible using
> default Flyway locations like db/migrations. You'll notice that I moved the
> current SQL scripts from its location to
> src/main/resources/kie-flyway/db/<module-name>/<db-type>
> 
> You can see an example of this file here (kie-flyway.properties
> <https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties>)
> and the new module locations here (locations
> <https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc>
> ).
> 
> 
> This addon supports the following configurations in the
> application.properties file:
> 
>    - kie.flyway.enabled=true|false # Enables the execution of the KIE
>    Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
>    and fully disabled in Spring-boot.
>    - kie.flyway.<module-name>.enabled=true|false # This is used to disable
>    specific module if we need to
> 
> 
> Some considerations:
> 
>    - This Add-On is intended to be used for development / test / examples
>    and not recommended to use in production
>    - As all our Add-Ons will use the default application datasource, so
>    it's not possible to make it use a different
>    - It doesn't cover all the Flyway features, it's only used to run
>    migrations to initialize our components.
>    - It doesn't replace the Platform Flyway in the kogito applications, the
>    user can still use Flyway to load its own scripts without having to care of
>    configuring anything from our modules.
> 
> 
> At this point, for simplicity I only tried to address the DB Initialization
> of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
> user-tasks), but this feature can be also brought to the Data-Index & Jobs
> Service if we want to, the only reason I didn't do it is to avoid doing
> changes that could break the Sonataflow Operator and the examples, but I'll
> happily apply the changes there if there's an agreement on this.
> 
> I think that's all... sorry for this long email but I think this
> improvement required some explanations.
> 
> Thank you!
> 
> Pere
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
+1

On Tue, Oct 1, 2024 at 9:56 AM Yeser Amer <ya...@apache.org> wrote:

> +1
>
> On 2024/10/01 07:37:30 Kennedy Bowers wrote:
> > +1
> >
> > On 2024/10/01 07:34:06 Jan Šťastný wrote:
> > > +1
> > >
> > > On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <
> gabriele.cardosi@gmail.com>
> > > > napísal(a):
> > > >
> > > > > +1
> > > > >
> > > > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > > > lightguardjp@apache.org> ha scritto:
> > > > >
> > > > > > +1 Great idea to make things easier for our users.
> > > > > >
> > > > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I recently rised some PR's to introduce a new KIE Flyway
> Add-On that
> > > > > > tries
> > > > > > > to solve some issues we found when enabling Flyway to
> initialize the
> > > > > > > application database in a compact setup (having runtime with
> other
> > > > > > modules
> > > > > > > that require persistence in the same app like data-index,
> > > > jobs-service
> > > > > > and
> > > > > > > data-audit). In this setup, a single Flyway instance is
> executed for
> > > > > the
> > > > > > > whole application during the startup and we are configuring it
> to
> > > > apply
> > > > > > all
> > > > > > > the SQL scripts (sorted by the version specified in the file
> name)
> > > > from
> > > > > > all
> > > > > > > those modules and keeping a single index for all the scripts
> applied
> > > > > > during
> > > > > > > the Flyway execution.
> > > > > > >
> > > > > > > The main problems we identified are related to the versioning
> of
> > > > those
> > > > > > SQL
> > > > > > > files inside the modules. Flyway gets script information
> (version,
> > > > > > > description...) from the file name (Ej:
> > > > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0)
> and
> > > > > expects
> > > > > > > version to be unique, it doesn't discriminate script source,
> > > > > description.
> > > > > > > Each of our add-on uses its own versions in the SQL scripts
> (ej:
> > > > > runtime
> > > > > > > persistence goes from V1.35.0 to V10.0.1, data-index from
> V1.32.0 to
> > > > > > > V1.45.0.4...), in this scenario it is very likely to have
> collisions
> > > > if
> > > > > > > different modules use the same version in the script name.
> > > > > > > Additionally, it's also possible that for the same reason, DB
> > > > upgrades
> > > > > > > won't work. For example if a module is updated to bring new SQL
> > > > Scripts
> > > > > > to
> > > > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > > > application
> > > > > > > running a Flyway again won't be able to apply those changes if
> in a
> > > > > > > previous run another add-on brought a script with a higher
> version
> > > > (in
> > > > > > this
> > > > > > > case runtime-persistence has v10).
> > > > > > >
> > > > > > > Another issue is that, in this setup, Flyway must be
> configured to
> > > > know
> > > > > > the
> > > > > > > locations of all the SQL scripts in the
> applications.properties. In
> > > > > > > general, this SQL files are in different locations, and there
> are
> > > > > modules
> > > > > > > that provide SQL's for different DB types (h2, pgsql). This
> can be a
> > > > > bit
> > > > > > > cumbersome to have it properly configured, specially for
> customers
> > > > that
> > > > > > are
> > > > > > > not familiar with the internals of our code.
> > > > > > >
> > > > > > >
> > > > > > > So, in order to help here, we are adding this Kie Flyway
> Add-On.
> > > > > > >
> > > > > > > The idea is that instead of using the default Flyway
> integration
> > > > > provided
> > > > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in
> charge
> > > > of
> > > > > > > running a separate instance of Flyway for each module that
> requires
> > > > it
> > > > > > and
> > > > > > > using an independent index for each one them, this way we can
> ensure
> > > > > that
> > > > > > > there will be no friction between SQL scripts. To make an
> module
> > > > > > available
> > > > > > > for KIE Flyway, first we must add the following dependency in
> our
> > > > > module
> > > > > > > pom.xml:
> > > > > > >
> > > > > > > <dependency>
> > > > > > >   <groupId>org.kie</groupId>
> > > > > > >
>  <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > > > </dependency>
> > > > > > >
> > > > > > > And make sure we include a kie-flyway.properties descriptor
> located
> > > > at
> > > > > `
> > > > > > > src/main/resources/META-INF` with the following properties:
> > > > > > >
> > > > > > > - module.name: logic name to identify the module.
> > > > > > > - module.locations.<db_type>: this property map is used to
> setup the
> > > > > > script
> > > > > > > locations per each database supported by our module, (ej:
> > > > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>).
> It's
> > > > > > > possible to use the module.locations.default key if we want to
> use
> > > > some
> > > > > > > fallback SQL's in case the application Database type isn't
> supported.
> > > > > > This
> > > > > > > locations should refer to a path (or comma-separate paths)
> unique to
> > > > > the
> > > > > > > module to avoid possible collisions and we should as much as
> possible
> > > > > > using
> > > > > > > default Flyway locations like db/migrations. You'll notice
> that I
> > > > moved
> > > > > > the
> > > > > > > current SQL scripts from its location to
> > > > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > > > >
> > > > > > > You can see an example of this file here (kie-flyway.properties
> > > > > > > <
> > > > > >
> > > > >
> > > >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > > > >)
> > > > > > > and the new module locations here (locations
> > > > > > > <
> > > > > >
> > > > >
> > > >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > > > >
> > > > > > > ).
> > > > > > >
> > > > > > >
> > > > > > > This addon supports the following configurations in the
> > > > > > > application.properties file:
> > > > > > >
> > > > > > >    - kie.flyway.enabled=true|false # Enables the execution of
> the KIE
> > > > > > >    Flyway Add-On on startup, it's disabled by default in prod
> mode
> > > > > > (Quarkus)
> > > > > > >    and fully disabled in Spring-boot.
> > > > > > >    - kie.flyway.<module-name>.enabled=true|false # This is
> used to
> > > > > > disable
> > > > > > >    specific module if we need to
> > > > > > >
> > > > > > >
> > > > > > > Some considerations:
> > > > > > >
> > > > > > >    - This Add-On is intended to be used for development / test
> /
> > > > > examples
> > > > > > >    and not recommended to use in production
> > > > > > >    - As all our Add-Ons will use the default application
> datasource,
> > > > so
> > > > > > >    it's not possible to make it use a different
> > > > > > >    - It doesn't cover all the Flyway features, it's only used
> to run
> > > > > > >    migrations to initialize our components.
> > > > > > >    - It doesn't replace the Platform Flyway in the kogito
> > > > applications,
> > > > > > the
> > > > > > >    user can still use Flyway to load its own scripts without
> having
> > > > to
> > > > > > care of
> > > > > > >    configuring anything from our modules.
> > > > > > >
> > > > > > >
> > > > > > > At this point, for simplicity I only tried to address the DB
> > > > > > Initialization
> > > > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > > > data-audit &
> > > > > > > user-tasks), but this feature can be also brought to the
> Data-Index &
> > > > > > Jobs
> > > > > > > Service if we want to, the only reason I didn't do it is to
> avoid
> > > > doing
> > > > > > > changes that could break the Sonataflow Operator and the
> examples,
> > > > but
> > > > > > I'll
> > > > > > > happily apply the changes there if there's an agreement on
> this.
> > > > > > >
> > > > > > > I think that's all... sorry for this long email but I think
> this
> > > > > > > improvement required some explanations.
> > > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > > > Pere
> > > > > > >
> > > > > >
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

Re: [HEADS UP] KIE Flyway Add-On

Posted by Yeser Amer <ya...@apache.org>.
+1

On 2024/10/01 07:37:30 Kennedy Bowers wrote:
> +1
> 
> On 2024/10/01 07:34:06 Jan Šťastný wrote:
> > +1
> > 
> > On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:
> > 
> > > +1
> > >
> > > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> > > napísal(a):
> > >
> > > > +1
> > > >
> > > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > > lightguardjp@apache.org> ha scritto:
> > > >
> > > > > +1 Great idea to make things easier for our users.
> > > > >
> > > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > > > tries
> > > > > > to solve some issues we found when enabling Flyway to initialize the
> > > > > > application database in a compact setup (having runtime with other
> > > > > modules
> > > > > > that require persistence in the same app like data-index,
> > > jobs-service
> > > > > and
> > > > > > data-audit). In this setup, a single Flyway instance is executed for
> > > > the
> > > > > > whole application during the startup and we are configuring it to
> > > apply
> > > > > all
> > > > > > the SQL scripts (sorted by the version specified in the file name)
> > > from
> > > > > all
> > > > > > those modules and keeping a single index for all the scripts applied
> > > > > during
> > > > > > the Flyway execution.
> > > > > >
> > > > > > The main problems we identified are related to the versioning of
> > > those
> > > > > SQL
> > > > > > files inside the modules. Flyway gets script information (version,
> > > > > > description...) from the file name (Ej:
> > > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > > > expects
> > > > > > version to be unique, it doesn't discriminate script source,
> > > > description.
> > > > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > > > runtime
> > > > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> > > if
> > > > > > different modules use the same version in the script name.
> > > > > > Additionally, it's also possible that for the same reason, DB
> > > upgrades
> > > > > > won't work. For example if a module is updated to bring new SQL
> > > Scripts
> > > > > to
> > > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > > application
> > > > > > running a Flyway again won't be able to apply those changes if in a
> > > > > > previous run another add-on brought a script with a higher version
> > > (in
> > > > > this
> > > > > > case runtime-persistence has v10).
> > > > > >
> > > > > > Another issue is that, in this setup, Flyway must be configured to
> > > know
> > > > > the
> > > > > > locations of all the SQL scripts in the applications.properties. In
> > > > > > general, this SQL files are in different locations, and there are
> > > > modules
> > > > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > > > bit
> > > > > > cumbersome to have it properly configured, specially for customers
> > > that
> > > > > are
> > > > > > not familiar with the internals of our code.
> > > > > >
> > > > > >
> > > > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > > > >
> > > > > > The idea is that instead of using the default Flyway integration
> > > > provided
> > > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> > > of
> > > > > > running a separate instance of Flyway for each module that requires
> > > it
> > > > > and
> > > > > > using an independent index for each one them, this way we can ensure
> > > > that
> > > > > > there will be no friction between SQL scripts. To make an module
> > > > > available
> > > > > > for KIE Flyway, first we must add the following dependency in our
> > > > module
> > > > > > pom.xml:
> > > > > >
> > > > > > <dependency>
> > > > > >   <groupId>org.kie</groupId>
> > > > > >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > > </dependency>
> > > > > >
> > > > > > And make sure we include a kie-flyway.properties descriptor located
> > > at
> > > > `
> > > > > > src/main/resources/META-INF` with the following properties:
> > > > > >
> > > > > > - module.name: logic name to identify the module.
> > > > > > - module.locations.<db_type>: this property map is used to setup the
> > > > > script
> > > > > > locations per each database supported by our module, (ej:
> > > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > > > possible to use the module.locations.default key if we want to use
> > > some
> > > > > > fallback SQL's in case the application Database type isn't supported.
> > > > > This
> > > > > > locations should refer to a path (or comma-separate paths) unique to
> > > > the
> > > > > > module to avoid possible collisions and we should as much as possible
> > > > > using
> > > > > > default Flyway locations like db/migrations. You'll notice that I
> > > moved
> > > > > the
> > > > > > current SQL scripts from its location to
> > > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > > >
> > > > > > You can see an example of this file here (kie-flyway.properties
> > > > > > <
> > > > >
> > > >
> > > https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > > >)
> > > > > > and the new module locations here (locations
> > > > > > <
> > > > >
> > > >
> > > https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > > >
> > > > > > ).
> > > > > >
> > > > > >
> > > > > > This addon supports the following configurations in the
> > > > > > application.properties file:
> > > > > >
> > > > > >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > > > >    Flyway Add-On on startup, it's disabled by default in prod mode
> > > > > (Quarkus)
> > > > > >    and fully disabled in Spring-boot.
> > > > > >    - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > > > disable
> > > > > >    specific module if we need to
> > > > > >
> > > > > >
> > > > > > Some considerations:
> > > > > >
> > > > > >    - This Add-On is intended to be used for development / test /
> > > > examples
> > > > > >    and not recommended to use in production
> > > > > >    - As all our Add-Ons will use the default application datasource,
> > > so
> > > > > >    it's not possible to make it use a different
> > > > > >    - It doesn't cover all the Flyway features, it's only used to run
> > > > > >    migrations to initialize our components.
> > > > > >    - It doesn't replace the Platform Flyway in the kogito
> > > applications,
> > > > > the
> > > > > >    user can still use Flyway to load its own scripts without having
> > > to
> > > > > care of
> > > > > >    configuring anything from our modules.
> > > > > >
> > > > > >
> > > > > > At this point, for simplicity I only tried to address the DB
> > > > > Initialization
> > > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > > data-audit &
> > > > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > > > Jobs
> > > > > > Service if we want to, the only reason I didn't do it is to avoid
> > > doing
> > > > > > changes that could break the Sonataflow Operator and the examples,
> > > but
> > > > > I'll
> > > > > > happily apply the changes there if there's an agreement on this.
> > > > > >
> > > > > > I think that's all... sorry for this long email but I think this
> > > > > > improvement required some explanations.
> > > > > >
> > > > > > Thank you!
> > > > > >
> > > > > > Pere
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > > >
> > >
> > 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Kennedy Bowers <kb...@apache.org>.
+1

On 2024/10/01 07:34:06 Jan Šťastný wrote:
> +1
> 
> On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:
> 
> > +1
> >
> > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> > napísal(a):
> >
> > > +1
> > >
> > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > lightguardjp@apache.org> ha scritto:
> > >
> > > > +1 Great idea to make things easier for our users.
> > > >
> > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > Hi,
> > > > >
> > > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > > tries
> > > > > to solve some issues we found when enabling Flyway to initialize the
> > > > > application database in a compact setup (having runtime with other
> > > > modules
> > > > > that require persistence in the same app like data-index,
> > jobs-service
> > > > and
> > > > > data-audit). In this setup, a single Flyway instance is executed for
> > > the
> > > > > whole application during the startup and we are configuring it to
> > apply
> > > > all
> > > > > the SQL scripts (sorted by the version specified in the file name)
> > from
> > > > all
> > > > > those modules and keeping a single index for all the scripts applied
> > > > during
> > > > > the Flyway execution.
> > > > >
> > > > > The main problems we identified are related to the versioning of
> > those
> > > > SQL
> > > > > files inside the modules. Flyway gets script information (version,
> > > > > description...) from the file name (Ej:
> > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > > expects
> > > > > version to be unique, it doesn't discriminate script source,
> > > description.
> > > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > > runtime
> > > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> > if
> > > > > different modules use the same version in the script name.
> > > > > Additionally, it's also possible that for the same reason, DB
> > upgrades
> > > > > won't work. For example if a module is updated to bring new SQL
> > Scripts
> > > > to
> > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > application
> > > > > running a Flyway again won't be able to apply those changes if in a
> > > > > previous run another add-on brought a script with a higher version
> > (in
> > > > this
> > > > > case runtime-persistence has v10).
> > > > >
> > > > > Another issue is that, in this setup, Flyway must be configured to
> > know
> > > > the
> > > > > locations of all the SQL scripts in the applications.properties. In
> > > > > general, this SQL files are in different locations, and there are
> > > modules
> > > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > > bit
> > > > > cumbersome to have it properly configured, specially for customers
> > that
> > > > are
> > > > > not familiar with the internals of our code.
> > > > >
> > > > >
> > > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > > >
> > > > > The idea is that instead of using the default Flyway integration
> > > provided
> > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> > of
> > > > > running a separate instance of Flyway for each module that requires
> > it
> > > > and
> > > > > using an independent index for each one them, this way we can ensure
> > > that
> > > > > there will be no friction between SQL scripts. To make an module
> > > > available
> > > > > for KIE Flyway, first we must add the following dependency in our
> > > module
> > > > > pom.xml:
> > > > >
> > > > > <dependency>
> > > > >   <groupId>org.kie</groupId>
> > > > >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > </dependency>
> > > > >
> > > > > And make sure we include a kie-flyway.properties descriptor located
> > at
> > > `
> > > > > src/main/resources/META-INF` with the following properties:
> > > > >
> > > > > - module.name: logic name to identify the module.
> > > > > - module.locations.<db_type>: this property map is used to setup the
> > > > script
> > > > > locations per each database supported by our module, (ej:
> > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > > possible to use the module.locations.default key if we want to use
> > some
> > > > > fallback SQL's in case the application Database type isn't supported.
> > > > This
> > > > > locations should refer to a path (or comma-separate paths) unique to
> > > the
> > > > > module to avoid possible collisions and we should as much as possible
> > > > using
> > > > > default Flyway locations like db/migrations. You'll notice that I
> > moved
> > > > the
> > > > > current SQL scripts from its location to
> > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > >
> > > > > You can see an example of this file here (kie-flyway.properties
> > > > > <
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > >)
> > > > > and the new module locations here (locations
> > > > > <
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > >
> > > > > ).
> > > > >
> > > > >
> > > > > This addon supports the following configurations in the
> > > > > application.properties file:
> > > > >
> > > > >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > > >    Flyway Add-On on startup, it's disabled by default in prod mode
> > > > (Quarkus)
> > > > >    and fully disabled in Spring-boot.
> > > > >    - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > > disable
> > > > >    specific module if we need to
> > > > >
> > > > >
> > > > > Some considerations:
> > > > >
> > > > >    - This Add-On is intended to be used for development / test /
> > > examples
> > > > >    and not recommended to use in production
> > > > >    - As all our Add-Ons will use the default application datasource,
> > so
> > > > >    it's not possible to make it use a different
> > > > >    - It doesn't cover all the Flyway features, it's only used to run
> > > > >    migrations to initialize our components.
> > > > >    - It doesn't replace the Platform Flyway in the kogito
> > applications,
> > > > the
> > > > >    user can still use Flyway to load its own scripts without having
> > to
> > > > care of
> > > > >    configuring anything from our modules.
> > > > >
> > > > >
> > > > > At this point, for simplicity I only tried to address the DB
> > > > Initialization
> > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > data-audit &
> > > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > > Jobs
> > > > > Service if we want to, the only reason I didn't do it is to avoid
> > doing
> > > > > changes that could break the Sonataflow Operator and the examples,
> > but
> > > > I'll
> > > > > happily apply the changes there if there's an agreement on this.
> > > > >
> > > > > I think that's all... sorry for this long email but I think this
> > > > > improvement required some explanations.
> > > > >
> > > > > Thank you!
> > > > >
> > > > > Pere
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [HEADS UP] KIE Flyway Add-On

Posted by Jan Šťastný <js...@apache.org>.
+1

On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:

> +1
>
> Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> napísal(a):
>
> > +1
> >
> > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > lightguardjp@apache.org> ha scritto:
> >
> > > +1 Great idea to make things easier for our users.
> > >
> > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > Hi,
> > > >
> > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > tries
> > > > to solve some issues we found when enabling Flyway to initialize the
> > > > application database in a compact setup (having runtime with other
> > > modules
> > > > that require persistence in the same app like data-index,
> jobs-service
> > > and
> > > > data-audit). In this setup, a single Flyway instance is executed for
> > the
> > > > whole application during the startup and we are configuring it to
> apply
> > > all
> > > > the SQL scripts (sorted by the version specified in the file name)
> from
> > > all
> > > > those modules and keeping a single index for all the scripts applied
> > > during
> > > > the Flyway execution.
> > > >
> > > > The main problems we identified are related to the versioning of
> those
> > > SQL
> > > > files inside the modules. Flyway gets script information (version,
> > > > description...) from the file name (Ej:
> > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > expects
> > > > version to be unique, it doesn't discriminate script source,
> > description.
> > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > runtime
> > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> if
> > > > different modules use the same version in the script name.
> > > > Additionally, it's also possible that for the same reason, DB
> upgrades
> > > > won't work. For example if a module is updated to bring new SQL
> Scripts
> > > to
> > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > application
> > > > running a Flyway again won't be able to apply those changes if in a
> > > > previous run another add-on brought a script with a higher version
> (in
> > > this
> > > > case runtime-persistence has v10).
> > > >
> > > > Another issue is that, in this setup, Flyway must be configured to
> know
> > > the
> > > > locations of all the SQL scripts in the applications.properties. In
> > > > general, this SQL files are in different locations, and there are
> > modules
> > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > bit
> > > > cumbersome to have it properly configured, specially for customers
> that
> > > are
> > > > not familiar with the internals of our code.
> > > >
> > > >
> > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > >
> > > > The idea is that instead of using the default Flyway integration
> > provided
> > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> of
> > > > running a separate instance of Flyway for each module that requires
> it
> > > and
> > > > using an independent index for each one them, this way we can ensure
> > that
> > > > there will be no friction between SQL scripts. To make an module
> > > available
> > > > for KIE Flyway, first we must add the following dependency in our
> > module
> > > > pom.xml:
> > > >
> > > > <dependency>
> > > >   <groupId>org.kie</groupId>
> > > >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > </dependency>
> > > >
> > > > And make sure we include a kie-flyway.properties descriptor located
> at
> > `
> > > > src/main/resources/META-INF` with the following properties:
> > > >
> > > > - module.name: logic name to identify the module.
> > > > - module.locations.<db_type>: this property map is used to setup the
> > > script
> > > > locations per each database supported by our module, (ej:
> > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > possible to use the module.locations.default key if we want to use
> some
> > > > fallback SQL's in case the application Database type isn't supported.
> > > This
> > > > locations should refer to a path (or comma-separate paths) unique to
> > the
> > > > module to avoid possible collisions and we should as much as possible
> > > using
> > > > default Flyway locations like db/migrations. You'll notice that I
> moved
> > > the
> > > > current SQL scripts from its location to
> > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > >
> > > > You can see an example of this file here (kie-flyway.properties
> > > > <
> > >
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > >)
> > > > and the new module locations here (locations
> > > > <
> > >
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > >
> > > > ).
> > > >
> > > >
> > > > This addon supports the following configurations in the
> > > > application.properties file:
> > > >
> > > >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > >    Flyway Add-On on startup, it's disabled by default in prod mode
> > > (Quarkus)
> > > >    and fully disabled in Spring-boot.
> > > >    - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > disable
> > > >    specific module if we need to
> > > >
> > > >
> > > > Some considerations:
> > > >
> > > >    - This Add-On is intended to be used for development / test /
> > examples
> > > >    and not recommended to use in production
> > > >    - As all our Add-Ons will use the default application datasource,
> so
> > > >    it's not possible to make it use a different
> > > >    - It doesn't cover all the Flyway features, it's only used to run
> > > >    migrations to initialize our components.
> > > >    - It doesn't replace the Platform Flyway in the kogito
> applications,
> > > the
> > > >    user can still use Flyway to load its own scripts without having
> to
> > > care of
> > > >    configuring anything from our modules.
> > > >
> > > >
> > > > At this point, for simplicity I only tried to address the DB
> > > Initialization
> > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > data-audit &
> > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > Jobs
> > > > Service if we want to, the only reason I didn't do it is to avoid
> doing
> > > > changes that could break the Sonataflow Operator and the examples,
> but
> > > I'll
> > > > happily apply the changes there if there's an agreement on this.
> > > >
> > > > I think that's all... sorry for this long email but I think this
> > > > improvement required some explanations.
> > > >
> > > > Thank you!
> > > >
> > > > Pere
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>

Re: [HEADS UP] KIE Flyway Add-On

Posted by Tibor Zimányi <tz...@apache.org>.
+1

Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
napísal(a):

> +1
>
> Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> lightguardjp@apache.org> ha scritto:
>
> > +1 Great idea to make things easier for our users.
> >
> > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > Hi,
> > >
> > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > tries
> > > to solve some issues we found when enabling Flyway to initialize the
> > > application database in a compact setup (having runtime with other
> > modules
> > > that require persistence in the same app like data-index, jobs-service
> > and
> > > data-audit). In this setup, a single Flyway instance is executed for
> the
> > > whole application during the startup and we are configuring it to apply
> > all
> > > the SQL scripts (sorted by the version specified in the file name) from
> > all
> > > those modules and keeping a single index for all the scripts applied
> > during
> > > the Flyway execution.
> > >
> > > The main problems we identified are related to the versioning of those
> > SQL
> > > files inside the modules. Flyway gets script information (version,
> > > description...) from the file name (Ej:
> > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> expects
> > > version to be unique, it doesn't discriminate script source,
> description.
> > > Each of our add-on uses its own versions in the SQL scripts (ej:
> runtime
> > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > > different modules use the same version in the script name.
> > > Additionally, it's also possible that for the same reason, DB upgrades
> > > won't work. For example if a module is updated to bring new SQL Scripts
> > to
> > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > application
> > > running a Flyway again won't be able to apply those changes if in a
> > > previous run another add-on brought a script with a higher version (in
> > this
> > > case runtime-persistence has v10).
> > >
> > > Another issue is that, in this setup, Flyway must be configured to know
> > the
> > > locations of all the SQL scripts in the applications.properties. In
> > > general, this SQL files are in different locations, and there are
> modules
> > > that provide SQL's for different DB types (h2, pgsql). This can be a
> bit
> > > cumbersome to have it properly configured, specially for customers that
> > are
> > > not familiar with the internals of our code.
> > >
> > >
> > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > >
> > > The idea is that instead of using the default Flyway integration
> provided
> > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > > running a separate instance of Flyway for each module that requires it
> > and
> > > using an independent index for each one them, this way we can ensure
> that
> > > there will be no friction between SQL scripts. To make an module
> > available
> > > for KIE Flyway, first we must add the following dependency in our
> module
> > > pom.xml:
> > >
> > > <dependency>
> > >   <groupId>org.kie</groupId>
> > >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > </dependency>
> > >
> > > And make sure we include a kie-flyway.properties descriptor located at
> `
> > > src/main/resources/META-INF` with the following properties:
> > >
> > > - module.name: logic name to identify the module.
> > > - module.locations.<db_type>: this property map is used to setup the
> > script
> > > locations per each database supported by our module, (ej:
> > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > possible to use the module.locations.default key if we want to use some
> > > fallback SQL's in case the application Database type isn't supported.
> > This
> > > locations should refer to a path (or comma-separate paths) unique to
> the
> > > module to avoid possible collisions and we should as much as possible
> > using
> > > default Flyway locations like db/migrations. You'll notice that I moved
> > the
> > > current SQL scripts from its location to
> > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > >
> > > You can see an example of this file here (kie-flyway.properties
> > > <
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > >)
> > > and the new module locations here (locations
> > > <
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > >
> > > ).
> > >
> > >
> > > This addon supports the following configurations in the
> > > application.properties file:
> > >
> > >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > >    Flyway Add-On on startup, it's disabled by default in prod mode
> > (Quarkus)
> > >    and fully disabled in Spring-boot.
> > >    - kie.flyway.<module-name>.enabled=true|false # This is used to
> > disable
> > >    specific module if we need to
> > >
> > >
> > > Some considerations:
> > >
> > >    - This Add-On is intended to be used for development / test /
> examples
> > >    and not recommended to use in production
> > >    - As all our Add-Ons will use the default application datasource, so
> > >    it's not possible to make it use a different
> > >    - It doesn't cover all the Flyway features, it's only used to run
> > >    migrations to initialize our components.
> > >    - It doesn't replace the Platform Flyway in the kogito applications,
> > the
> > >    user can still use Flyway to load its own scripts without having to
> > care of
> > >    configuring anything from our modules.
> > >
> > >
> > > At this point, for simplicity I only tried to address the DB
> > Initialization
> > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > data-audit &
> > > user-tasks), but this feature can be also brought to the Data-Index &
> > Jobs
> > > Service if we want to, the only reason I didn't do it is to avoid doing
> > > changes that could break the Sonataflow Operator and the examples, but
> > I'll
> > > happily apply the changes there if there's an agreement on this.
> > >
> > > I think that's all... sorry for this long email but I think this
> > > improvement required some explanations.
> > >
> > > Thank you!
> > >
> > > Pere
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>

Re: [HEADS UP] KIE Flyway Add-On

Posted by Gabriele Cardosi <ga...@gmail.com>.
+1

Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
lightguardjp@apache.org> ha scritto:

> +1 Great idea to make things easier for our users.
>
> On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > Hi,
> >
> > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> tries
> > to solve some issues we found when enabling Flyway to initialize the
> > application database in a compact setup (having runtime with other
> modules
> > that require persistence in the same app like data-index, jobs-service
> and
> > data-audit). In this setup, a single Flyway instance is executed for the
> > whole application during the startup and we are configuring it to apply
> all
> > the SQL scripts (sorted by the version specified in the file name) from
> all
> > those modules and keeping a single index for all the scripts applied
> during
> > the Flyway execution.
> >
> > The main problems we identified are related to the versioning of those
> SQL
> > files inside the modules. Flyway gets script information (version,
> > description...) from the file name (Ej:
> > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> > version to be unique, it doesn't discriminate script source, description.
> > Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > different modules use the same version in the script name.
> > Additionally, it's also possible that for the same reason, DB upgrades
> > won't work. For example if a module is updated to bring new SQL Scripts
> to
> > upgrade the DB (let's say data-index bringing V1.5.0) the same
> application
> > running a Flyway again won't be able to apply those changes if in a
> > previous run another add-on brought a script with a higher version (in
> this
> > case runtime-persistence has v10).
> >
> > Another issue is that, in this setup, Flyway must be configured to know
> the
> > locations of all the SQL scripts in the applications.properties. In
> > general, this SQL files are in different locations, and there are modules
> > that provide SQL's for different DB types (h2, pgsql). This can be a bit
> > cumbersome to have it properly configured, specially for customers that
> are
> > not familiar with the internals of our code.
> >
> >
> > So, in order to help here, we are adding this Kie Flyway Add-On.
> >
> > The idea is that instead of using the default Flyway integration provided
> > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > running a separate instance of Flyway for each module that requires it
> and
> > using an independent index for each one them, this way we can ensure that
> > there will be no friction between SQL scripts. To make an module
> available
> > for KIE Flyway, first we must add the following dependency in our module
> > pom.xml:
> >
> > <dependency>
> >   <groupId>org.kie</groupId>
> >   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > </dependency>
> >
> > And make sure we include a kie-flyway.properties descriptor located at `
> > src/main/resources/META-INF` with the following properties:
> >
> > - module.name: logic name to identify the module.
> > - module.locations.<db_type>: this property map is used to setup the
> script
> > locations per each database supported by our module, (ej:
> > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > possible to use the module.locations.default key if we want to use some
> > fallback SQL's in case the application Database type isn't supported.
> This
> > locations should refer to a path (or comma-separate paths) unique to the
> > module to avoid possible collisions and we should as much as possible
> using
> > default Flyway locations like db/migrations. You'll notice that I moved
> the
> > current SQL scripts from its location to
> > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> >
> > You can see an example of this file here (kie-flyway.properties
> > <
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> >)
> > and the new module locations here (locations
> > <
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> >
> > ).
> >
> >
> > This addon supports the following configurations in the
> > application.properties file:
> >
> >    - kie.flyway.enabled=true|false # Enables the execution of the KIE
> >    Flyway Add-On on startup, it's disabled by default in prod mode
> (Quarkus)
> >    and fully disabled in Spring-boot.
> >    - kie.flyway.<module-name>.enabled=true|false # This is used to
> disable
> >    specific module if we need to
> >
> >
> > Some considerations:
> >
> >    - This Add-On is intended to be used for development / test / examples
> >    and not recommended to use in production
> >    - As all our Add-Ons will use the default application datasource, so
> >    it's not possible to make it use a different
> >    - It doesn't cover all the Flyway features, it's only used to run
> >    migrations to initialize our components.
> >    - It doesn't replace the Platform Flyway in the kogito applications,
> the
> >    user can still use Flyway to load its own scripts without having to
> care of
> >    configuring anything from our modules.
> >
> >
> > At this point, for simplicity I only tried to address the DB
> Initialization
> > of our Add-Ons (runtime persistence, data-index, jobs-service,
> data-audit &
> > user-tasks), but this feature can be also brought to the Data-Index &
> Jobs
> > Service if we want to, the only reason I didn't do it is to avoid doing
> > changes that could break the Sonataflow Operator and the examples, but
> I'll
> > happily apply the changes there if there's an agreement on this.
> >
> > I think that's all... sorry for this long email but I think this
> > improvement required some explanations.
> >
> > Thank you!
> >
> > Pere
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

Re: [HEADS UP] KIE Flyway Add-On

Posted by Jason Porter <li...@apache.org>.
+1 Great idea to make things easier for our users.

On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> Hi,
> 
> I recently rised some PR's to introduce a new KIE Flyway Add-On that tries
> to solve some issues we found when enabling Flyway to initialize the
> application database in a compact setup (having runtime with other modules
> that require persistence in the same app like data-index, jobs-service and
> data-audit). In this setup, a single Flyway instance is executed for the
> whole application during the startup and we are configuring it to apply all
> the SQL scripts (sorted by the version specified in the file name) from all
> those modules and keeping a single index for all the scripts applied during
> the Flyway execution.
> 
> The main problems we identified are related to the versioning of those SQL
> files inside the modules. Flyway gets script information (version,
> description...) from the file name (Ej:
> V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> version to be unique, it doesn't discriminate script source, description.
> Each of our add-on uses its own versions in the SQL scripts (ej:  runtime
> persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> V1.45.0.4...), in this scenario it is very likely to have collisions if
> different modules use the same version in the script name.
> Additionally, it's also possible that for the same reason, DB upgrades
> won't work. For example if a module is updated to bring new SQL Scripts to
> upgrade the DB (let's say data-index bringing V1.5.0) the same application
> running a Flyway again won't be able to apply those changes if in a
> previous run another add-on brought a script with a higher version (in this
> case runtime-persistence has v10).
> 
> Another issue is that, in this setup, Flyway must be configured to know the
> locations of all the SQL scripts in the applications.properties. In
> general, this SQL files are in different locations, and there are modules
> that provide SQL's for different DB types (h2, pgsql). This can be a bit
> cumbersome to have it properly configured, specially for customers that are
> not familiar with the internals of our code.
> 
> 
> So, in order to help here, we are adding this Kie Flyway Add-On.
> 
> The idea is that instead of using the default Flyway integration provided
> by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> running a separate instance of Flyway for each module that requires it and
> using an independent index for each one them, this way we can ensure that
> there will be no friction between SQL scripts. To make an module available
> for KIE Flyway, first we must add the following dependency in our module
> pom.xml:
> 
> <dependency>
>   <groupId>org.kie</groupId>
>   <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> </dependency>
> 
> And make sure we include a kie-flyway.properties descriptor located at `
> src/main/resources/META-INF` with the following properties:
> 
> - module.name: logic name to identify the module.
> - module.locations.<db_type>: this property map is used to setup the script
> locations per each database supported by our module, (ej:
> module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> possible to use the module.locations.default key if we want to use some
> fallback SQL's in case the application Database type isn't supported. This
> locations should refer to a path (or comma-separate paths) unique to the
> module to avoid possible collisions and we should as much as possible using
> default Flyway locations like db/migrations. You'll notice that I moved the
> current SQL scripts from its location to
> src/main/resources/kie-flyway/db/<module-name>/<db-type>
> 
> You can see an example of this file here (kie-flyway.properties
> <https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties>)
> and the new module locations here (locations
> <https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc>
> ).
> 
> 
> This addon supports the following configurations in the
> application.properties file:
> 
>    - kie.flyway.enabled=true|false # Enables the execution of the KIE
>    Flyway Add-On on startup, it's disabled by default in prod mode (Quarkus)
>    and fully disabled in Spring-boot.
>    - kie.flyway.<module-name>.enabled=true|false # This is used to disable
>    specific module if we need to
> 
> 
> Some considerations:
> 
>    - This Add-On is intended to be used for development / test / examples
>    and not recommended to use in production
>    - As all our Add-Ons will use the default application datasource, so
>    it's not possible to make it use a different
>    - It doesn't cover all the Flyway features, it's only used to run
>    migrations to initialize our components.
>    - It doesn't replace the Platform Flyway in the kogito applications, the
>    user can still use Flyway to load its own scripts without having to care of
>    configuring anything from our modules.
> 
> 
> At this point, for simplicity I only tried to address the DB Initialization
> of our Add-Ons (runtime persistence, data-index, jobs-service, data-audit &
> user-tasks), but this feature can be also brought to the Data-Index & Jobs
> Service if we want to, the only reason I didn't do it is to avoid doing
> changes that could break the Sonataflow Operator and the examples, but I'll
> happily apply the changes there if there's an agreement on this.
> 
> I think that's all... sorry for this long email but I think this
> improvement required some explanations.
> 
> Thank you!
> 
> Pere
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org