You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kie.apache.org by Gabriele Cardosi <ga...@gmail.com> on 2024/10/01 06:28:48 UTC
Re: [HEADS UP] KIE Flyway Add-On
+1
Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
lightguardjp@apache.org> ha scritto:
> +1 Great idea to make things easier for our users.
>
> On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > Hi,
> >
> > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> tries
> > to solve some issues we found when enabling Flyway to initialize the
> > application database in a compact setup (having runtime with other
> modules
> > that require persistence in the same app like data-index, jobs-service
> and
> > data-audit). In this setup, a single Flyway instance is executed for the
> > whole application during the startup and we are configuring it to apply
> all
> > the SQL scripts (sorted by the version specified in the file name) from
> all
> > those modules and keeping a single index for all the scripts applied
> during
> > the Flyway execution.
> >
> > The main problems we identified are related to the versioning of those
> SQL
> > files inside the modules. Flyway gets script information (version,
> > description...) from the file name (Ej:
> > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and expects
> > version to be unique, it doesn't discriminate script source, description.
> > Each of our add-on uses its own versions in the SQL scripts (ej: runtime
> > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > different modules use the same version in the script name.
> > Additionally, it's also possible that for the same reason, DB upgrades
> > won't work. For example if a module is updated to bring new SQL Scripts
> to
> > upgrade the DB (let's say data-index bringing V1.5.0) the same
> application
> > running a Flyway again won't be able to apply those changes if in a
> > previous run another add-on brought a script with a higher version (in
> this
> > case runtime-persistence has v10).
> >
> > Another issue is that, in this setup, Flyway must be configured to know
> the
> > locations of all the SQL scripts in the applications.properties. In
> > general, this SQL files are in different locations, and there are modules
> > that provide SQL's for different DB types (h2, pgsql). This can be a bit
> > cumbersome to have it properly configured, specially for customers that
> are
> > not familiar with the internals of our code.
> >
> >
> > So, in order to help here, we are adding this Kie Flyway Add-On.
> >
> > The idea is that instead of using the default Flyway integration provided
> > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > running a separate instance of Flyway for each module that requires it
> and
> > using an independent index for each one them, this way we can ensure that
> > there will be no friction between SQL scripts. To make an module
> available
> > for KIE Flyway, first we must add the following dependency in our module
> > pom.xml:
> >
> > <dependency>
> > <groupId>org.kie</groupId>
> > <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > </dependency>
> >
> > And make sure we include a kie-flyway.properties descriptor located at `
> > src/main/resources/META-INF` with the following properties:
> >
> > - module.name: logic name to identify the module.
> > - module.locations.<db_type>: this property map is used to setup the
> script
> > locations per each database supported by our module, (ej:
> > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > possible to use the module.locations.default key if we want to use some
> > fallback SQL's in case the application Database type isn't supported.
> This
> > locations should refer to a path (or comma-separate paths) unique to the
> > module to avoid possible collisions and we should as much as possible
> using
> > default Flyway locations like db/migrations. You'll notice that I moved
> the
> > current SQL scripts from its location to
> > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> >
> > You can see an example of this file here (kie-flyway.properties
> > <
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> >)
> > and the new module locations here (locations
> > <
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> >
> > ).
> >
> >
> > This addon supports the following configurations in the
> > application.properties file:
> >
> > - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > Flyway Add-On on startup, it's disabled by default in prod mode
> (Quarkus)
> > and fully disabled in Spring-boot.
> > - kie.flyway.<module-name>.enabled=true|false # This is used to
> disable
> > specific module if we need to
> >
> >
> > Some considerations:
> >
> > - This Add-On is intended to be used for development / test / examples
> > and not recommended to use in production
> > - As all our Add-Ons will use the default application datasource, so
> > it's not possible to make it use a different
> > - It doesn't cover all the Flyway features, it's only used to run
> > migrations to initialize our components.
> > - It doesn't replace the Platform Flyway in the kogito applications,
> the
> > user can still use Flyway to load its own scripts without having to
> care of
> > configuring anything from our modules.
> >
> >
> > At this point, for simplicity I only tried to address the DB
> Initialization
> > of our Add-Ons (runtime persistence, data-index, jobs-service,
> data-audit &
> > user-tasks), but this feature can be also brought to the Data-Index &
> Jobs
> > Service if we want to, the only reason I didn't do it is to avoid doing
> > changes that could break the Sonataflow Operator and the examples, but
> I'll
> > happily apply the changes there if there's an agreement on this.
> >
> > I think that's all... sorry for this long email but I think this
> > improvement required some explanations.
> >
> > Thank you!
> >
> > Pere
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>
Re: [HEADS UP] KIE Flyway Add-On
Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
+1
On Tue, Oct 1, 2024 at 9:56 AM Yeser Amer <ya...@apache.org> wrote:
> +1
>
> On 2024/10/01 07:37:30 Kennedy Bowers wrote:
> > +1
> >
> > On 2024/10/01 07:34:06 Jan Šťastný wrote:
> > > +1
> > >
> > > On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <
> gabriele.cardosi@gmail.com>
> > > > napísal(a):
> > > >
> > > > > +1
> > > > >
> > > > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > > > lightguardjp@apache.org> ha scritto:
> > > > >
> > > > > > +1 Great idea to make things easier for our users.
> > > > > >
> > > > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I recently rised some PR's to introduce a new KIE Flyway
> Add-On that
> > > > > > tries
> > > > > > > to solve some issues we found when enabling Flyway to
> initialize the
> > > > > > > application database in a compact setup (having runtime with
> other
> > > > > > modules
> > > > > > > that require persistence in the same app like data-index,
> > > > jobs-service
> > > > > > and
> > > > > > > data-audit). In this setup, a single Flyway instance is
> executed for
> > > > > the
> > > > > > > whole application during the startup and we are configuring it
> to
> > > > apply
> > > > > > all
> > > > > > > the SQL scripts (sorted by the version specified in the file
> name)
> > > > from
> > > > > > all
> > > > > > > those modules and keeping a single index for all the scripts
> applied
> > > > > > during
> > > > > > > the Flyway execution.
> > > > > > >
> > > > > > > The main problems we identified are related to the versioning
> of
> > > > those
> > > > > > SQL
> > > > > > > files inside the modules. Flyway gets script information
> (version,
> > > > > > > description...) from the file name (Ej:
> > > > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0)
> and
> > > > > expects
> > > > > > > version to be unique, it doesn't discriminate script source,
> > > > > description.
> > > > > > > Each of our add-on uses its own versions in the SQL scripts
> (ej:
> > > > > runtime
> > > > > > > persistence goes from V1.35.0 to V10.0.1, data-index from
> V1.32.0 to
> > > > > > > V1.45.0.4...), in this scenario it is very likely to have
> collisions
> > > > if
> > > > > > > different modules use the same version in the script name.
> > > > > > > Additionally, it's also possible that for the same reason, DB
> > > > upgrades
> > > > > > > won't work. For example if a module is updated to bring new SQL
> > > > Scripts
> > > > > > to
> > > > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > > > application
> > > > > > > running a Flyway again won't be able to apply those changes if
> in a
> > > > > > > previous run another add-on brought a script with a higher
> version
> > > > (in
> > > > > > this
> > > > > > > case runtime-persistence has v10).
> > > > > > >
> > > > > > > Another issue is that, in this setup, Flyway must be
> configured to
> > > > know
> > > > > > the
> > > > > > > locations of all the SQL scripts in the
> applications.properties. In
> > > > > > > general, this SQL files are in different locations, and there
> are
> > > > > modules
> > > > > > > that provide SQL's for different DB types (h2, pgsql). This
> can be a
> > > > > bit
> > > > > > > cumbersome to have it properly configured, specially for
> customers
> > > > that
> > > > > > are
> > > > > > > not familiar with the internals of our code.
> > > > > > >
> > > > > > >
> > > > > > > So, in order to help here, we are adding this Kie Flyway
> Add-On.
> > > > > > >
> > > > > > > The idea is that instead of using the default Flyway
> integration
> > > > > provided
> > > > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in
> charge
> > > > of
> > > > > > > running a separate instance of Flyway for each module that
> requires
> > > > it
> > > > > > and
> > > > > > > using an independent index for each one them, this way we can
> ensure
> > > > > that
> > > > > > > there will be no friction between SQL scripts. To make an
> module
> > > > > > available
> > > > > > > for KIE Flyway, first we must add the following dependency in
> our
> > > > > module
> > > > > > > pom.xml:
> > > > > > >
> > > > > > > <dependency>
> > > > > > > <groupId>org.kie</groupId>
> > > > > > >
> <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > > > </dependency>
> > > > > > >
> > > > > > > And make sure we include a kie-flyway.properties descriptor
> located
> > > > at
> > > > > `
> > > > > > > src/main/resources/META-INF` with the following properties:
> > > > > > >
> > > > > > > - module.name: logic name to identify the module.
> > > > > > > - module.locations.<db_type>: this property map is used to
> setup the
> > > > > > script
> > > > > > > locations per each database supported by our module, (ej:
> > > > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>).
> It's
> > > > > > > possible to use the module.locations.default key if we want to
> use
> > > > some
> > > > > > > fallback SQL's in case the application Database type isn't
> supported.
> > > > > > This
> > > > > > > locations should refer to a path (or comma-separate paths)
> unique to
> > > > > the
> > > > > > > module to avoid possible collisions and we should as much as
> possible
> > > > > > using
> > > > > > > default Flyway locations like db/migrations. You'll notice
> that I
> > > > moved
> > > > > > the
> > > > > > > current SQL scripts from its location to
> > > > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > > > >
> > > > > > > You can see an example of this file here (kie-flyway.properties
> > > > > > > <
> > > > > >
> > > > >
> > > >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > > > >)
> > > > > > > and the new module locations here (locations
> > > > > > > <
> > > > > >
> > > > >
> > > >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > > > >
> > > > > > > ).
> > > > > > >
> > > > > > >
> > > > > > > This addon supports the following configurations in the
> > > > > > > application.properties file:
> > > > > > >
> > > > > > > - kie.flyway.enabled=true|false # Enables the execution of
> the KIE
> > > > > > > Flyway Add-On on startup, it's disabled by default in prod
> mode
> > > > > > (Quarkus)
> > > > > > > and fully disabled in Spring-boot.
> > > > > > > - kie.flyway.<module-name>.enabled=true|false # This is
> used to
> > > > > > disable
> > > > > > > specific module if we need to
> > > > > > >
> > > > > > >
> > > > > > > Some considerations:
> > > > > > >
> > > > > > > - This Add-On is intended to be used for development / test
> /
> > > > > examples
> > > > > > > and not recommended to use in production
> > > > > > > - As all our Add-Ons will use the default application
> datasource,
> > > > so
> > > > > > > it's not possible to make it use a different
> > > > > > > - It doesn't cover all the Flyway features, it's only used
> to run
> > > > > > > migrations to initialize our components.
> > > > > > > - It doesn't replace the Platform Flyway in the kogito
> > > > applications,
> > > > > > the
> > > > > > > user can still use Flyway to load its own scripts without
> having
> > > > to
> > > > > > care of
> > > > > > > configuring anything from our modules.
> > > > > > >
> > > > > > >
> > > > > > > At this point, for simplicity I only tried to address the DB
> > > > > > Initialization
> > > > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > > > data-audit &
> > > > > > > user-tasks), but this feature can be also brought to the
> Data-Index &
> > > > > > Jobs
> > > > > > > Service if we want to, the only reason I didn't do it is to
> avoid
> > > > doing
> > > > > > > changes that could break the Sonataflow Operator and the
> examples,
> > > > but
> > > > > > I'll
> > > > > > > happily apply the changes there if there's an agreement on
> this.
> > > > > > >
> > > > > > > I think that's all... sorry for this long email but I think
> this
> > > > > > > improvement required some explanations.
> > > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > > > Pere
> > > > > > >
> > > > > >
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>
Re: [HEADS UP] KIE Flyway Add-On
Posted by Yeser Amer <ya...@apache.org>.
+1
On 2024/10/01 07:37:30 Kennedy Bowers wrote:
> +1
>
> On 2024/10/01 07:34:06 Jan Šťastný wrote:
> > +1
> >
> > On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:
> >
> > > +1
> > >
> > > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> > > napísal(a):
> > >
> > > > +1
> > > >
> > > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > > lightguardjp@apache.org> ha scritto:
> > > >
> > > > > +1 Great idea to make things easier for our users.
> > > > >
> > > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > > > tries
> > > > > > to solve some issues we found when enabling Flyway to initialize the
> > > > > > application database in a compact setup (having runtime with other
> > > > > modules
> > > > > > that require persistence in the same app like data-index,
> > > jobs-service
> > > > > and
> > > > > > data-audit). In this setup, a single Flyway instance is executed for
> > > > the
> > > > > > whole application during the startup and we are configuring it to
> > > apply
> > > > > all
> > > > > > the SQL scripts (sorted by the version specified in the file name)
> > > from
> > > > > all
> > > > > > those modules and keeping a single index for all the scripts applied
> > > > > during
> > > > > > the Flyway execution.
> > > > > >
> > > > > > The main problems we identified are related to the versioning of
> > > those
> > > > > SQL
> > > > > > files inside the modules. Flyway gets script information (version,
> > > > > > description...) from the file name (Ej:
> > > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > > > expects
> > > > > > version to be unique, it doesn't discriminate script source,
> > > > description.
> > > > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > > > runtime
> > > > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> > > if
> > > > > > different modules use the same version in the script name.
> > > > > > Additionally, it's also possible that for the same reason, DB
> > > upgrades
> > > > > > won't work. For example if a module is updated to bring new SQL
> > > Scripts
> > > > > to
> > > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > > application
> > > > > > running a Flyway again won't be able to apply those changes if in a
> > > > > > previous run another add-on brought a script with a higher version
> > > (in
> > > > > this
> > > > > > case runtime-persistence has v10).
> > > > > >
> > > > > > Another issue is that, in this setup, Flyway must be configured to
> > > know
> > > > > the
> > > > > > locations of all the SQL scripts in the applications.properties. In
> > > > > > general, this SQL files are in different locations, and there are
> > > > modules
> > > > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > > > bit
> > > > > > cumbersome to have it properly configured, specially for customers
> > > that
> > > > > are
> > > > > > not familiar with the internals of our code.
> > > > > >
> > > > > >
> > > > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > > > >
> > > > > > The idea is that instead of using the default Flyway integration
> > > > provided
> > > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> > > of
> > > > > > running a separate instance of Flyway for each module that requires
> > > it
> > > > > and
> > > > > > using an independent index for each one them, this way we can ensure
> > > > that
> > > > > > there will be no friction between SQL scripts. To make an module
> > > > > available
> > > > > > for KIE Flyway, first we must add the following dependency in our
> > > > module
> > > > > > pom.xml:
> > > > > >
> > > > > > <dependency>
> > > > > > <groupId>org.kie</groupId>
> > > > > > <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > > </dependency>
> > > > > >
> > > > > > And make sure we include a kie-flyway.properties descriptor located
> > > at
> > > > `
> > > > > > src/main/resources/META-INF` with the following properties:
> > > > > >
> > > > > > - module.name: logic name to identify the module.
> > > > > > - module.locations.<db_type>: this property map is used to setup the
> > > > > script
> > > > > > locations per each database supported by our module, (ej:
> > > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > > > possible to use the module.locations.default key if we want to use
> > > some
> > > > > > fallback SQL's in case the application Database type isn't supported.
> > > > > This
> > > > > > locations should refer to a path (or comma-separate paths) unique to
> > > > the
> > > > > > module to avoid possible collisions and we should as much as possible
> > > > > using
> > > > > > default Flyway locations like db/migrations. You'll notice that I
> > > moved
> > > > > the
> > > > > > current SQL scripts from its location to
> > > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > > >
> > > > > > You can see an example of this file here (kie-flyway.properties
> > > > > > <
> > > > >
> > > >
> > > https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > > >)
> > > > > > and the new module locations here (locations
> > > > > > <
> > > > >
> > > >
> > > https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > > >
> > > > > > ).
> > > > > >
> > > > > >
> > > > > > This addon supports the following configurations in the
> > > > > > application.properties file:
> > > > > >
> > > > > > - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > > > > Flyway Add-On on startup, it's disabled by default in prod mode
> > > > > (Quarkus)
> > > > > > and fully disabled in Spring-boot.
> > > > > > - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > > > disable
> > > > > > specific module if we need to
> > > > > >
> > > > > >
> > > > > > Some considerations:
> > > > > >
> > > > > > - This Add-On is intended to be used for development / test /
> > > > examples
> > > > > > and not recommended to use in production
> > > > > > - As all our Add-Ons will use the default application datasource,
> > > so
> > > > > > it's not possible to make it use a different
> > > > > > - It doesn't cover all the Flyway features, it's only used to run
> > > > > > migrations to initialize our components.
> > > > > > - It doesn't replace the Platform Flyway in the kogito
> > > applications,
> > > > > the
> > > > > > user can still use Flyway to load its own scripts without having
> > > to
> > > > > care of
> > > > > > configuring anything from our modules.
> > > > > >
> > > > > >
> > > > > > At this point, for simplicity I only tried to address the DB
> > > > > Initialization
> > > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > > data-audit &
> > > > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > > > Jobs
> > > > > > Service if we want to, the only reason I didn't do it is to avoid
> > > doing
> > > > > > changes that could break the Sonataflow Operator and the examples,
> > > but
> > > > > I'll
> > > > > > happily apply the changes there if there's an agreement on this.
> > > > > >
> > > > > > I think that's all... sorry for this long email but I think this
> > > > > > improvement required some explanations.
> > > > > >
> > > > > > Thank you!
> > > > > >
> > > > > > Pere
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org
Re: [HEADS UP] KIE Flyway Add-On
Posted by Kennedy Bowers <kb...@apache.org>.
+1
On 2024/10/01 07:34:06 Jan Šťastný wrote:
> +1
>
> On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:
>
> > +1
> >
> > Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> > napísal(a):
> >
> > > +1
> > >
> > > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > > lightguardjp@apache.org> ha scritto:
> > >
> > > > +1 Great idea to make things easier for our users.
> > > >
> > > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > > Hi,
> > > > >
> > > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > > tries
> > > > > to solve some issues we found when enabling Flyway to initialize the
> > > > > application database in a compact setup (having runtime with other
> > > > modules
> > > > > that require persistence in the same app like data-index,
> > jobs-service
> > > > and
> > > > > data-audit). In this setup, a single Flyway instance is executed for
> > > the
> > > > > whole application during the startup and we are configuring it to
> > apply
> > > > all
> > > > > the SQL scripts (sorted by the version specified in the file name)
> > from
> > > > all
> > > > > those modules and keeping a single index for all the scripts applied
> > > > during
> > > > > the Flyway execution.
> > > > >
> > > > > The main problems we identified are related to the versioning of
> > those
> > > > SQL
> > > > > files inside the modules. Flyway gets script information (version,
> > > > > description...) from the file name (Ej:
> > > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > > expects
> > > > > version to be unique, it doesn't discriminate script source,
> > > description.
> > > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > > runtime
> > > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> > if
> > > > > different modules use the same version in the script name.
> > > > > Additionally, it's also possible that for the same reason, DB
> > upgrades
> > > > > won't work. For example if a module is updated to bring new SQL
> > Scripts
> > > > to
> > > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > > application
> > > > > running a Flyway again won't be able to apply those changes if in a
> > > > > previous run another add-on brought a script with a higher version
> > (in
> > > > this
> > > > > case runtime-persistence has v10).
> > > > >
> > > > > Another issue is that, in this setup, Flyway must be configured to
> > know
> > > > the
> > > > > locations of all the SQL scripts in the applications.properties. In
> > > > > general, this SQL files are in different locations, and there are
> > > modules
> > > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > > bit
> > > > > cumbersome to have it properly configured, specially for customers
> > that
> > > > are
> > > > > not familiar with the internals of our code.
> > > > >
> > > > >
> > > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > > >
> > > > > The idea is that instead of using the default Flyway integration
> > > provided
> > > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> > of
> > > > > running a separate instance of Flyway for each module that requires
> > it
> > > > and
> > > > > using an independent index for each one them, this way we can ensure
> > > that
> > > > > there will be no friction between SQL scripts. To make an module
> > > > available
> > > > > for KIE Flyway, first we must add the following dependency in our
> > > module
> > > > > pom.xml:
> > > > >
> > > > > <dependency>
> > > > > <groupId>org.kie</groupId>
> > > > > <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > > </dependency>
> > > > >
> > > > > And make sure we include a kie-flyway.properties descriptor located
> > at
> > > `
> > > > > src/main/resources/META-INF` with the following properties:
> > > > >
> > > > > - module.name: logic name to identify the module.
> > > > > - module.locations.<db_type>: this property map is used to setup the
> > > > script
> > > > > locations per each database supported by our module, (ej:
> > > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > > possible to use the module.locations.default key if we want to use
> > some
> > > > > fallback SQL's in case the application Database type isn't supported.
> > > > This
> > > > > locations should refer to a path (or comma-separate paths) unique to
> > > the
> > > > > module to avoid possible collisions and we should as much as possible
> > > > using
> > > > > default Flyway locations like db/migrations. You'll notice that I
> > moved
> > > > the
> > > > > current SQL scripts from its location to
> > > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > > >
> > > > > You can see an example of this file here (kie-flyway.properties
> > > > > <
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > > >)
> > > > > and the new module locations here (locations
> > > > > <
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > > >
> > > > > ).
> > > > >
> > > > >
> > > > > This addon supports the following configurations in the
> > > > > application.properties file:
> > > > >
> > > > > - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > > > Flyway Add-On on startup, it's disabled by default in prod mode
> > > > (Quarkus)
> > > > > and fully disabled in Spring-boot.
> > > > > - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > > disable
> > > > > specific module if we need to
> > > > >
> > > > >
> > > > > Some considerations:
> > > > >
> > > > > - This Add-On is intended to be used for development / test /
> > > examples
> > > > > and not recommended to use in production
> > > > > - As all our Add-Ons will use the default application datasource,
> > so
> > > > > it's not possible to make it use a different
> > > > > - It doesn't cover all the Flyway features, it's only used to run
> > > > > migrations to initialize our components.
> > > > > - It doesn't replace the Platform Flyway in the kogito
> > applications,
> > > > the
> > > > > user can still use Flyway to load its own scripts without having
> > to
> > > > care of
> > > > > configuring anything from our modules.
> > > > >
> > > > >
> > > > > At this point, for simplicity I only tried to address the DB
> > > > Initialization
> > > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > > data-audit &
> > > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > > Jobs
> > > > > Service if we want to, the only reason I didn't do it is to avoid
> > doing
> > > > > changes that could break the Sonataflow Operator and the examples,
> > but
> > > > I'll
> > > > > happily apply the changes there if there's an agreement on this.
> > > > >
> > > > > I think that's all... sorry for this long email but I think this
> > > > > improvement required some explanations.
> > > > >
> > > > > Thank you!
> > > > >
> > > > > Pere
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org
Re: [HEADS UP] KIE Flyway Add-On
Posted by Jan Šťastný <js...@apache.org>.
+1
On Tue, 1 Oct 2024 at 09:24, Tibor Zimányi <tz...@apache.org> wrote:
> +1
>
> Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
> napísal(a):
>
> > +1
> >
> > Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> > lightguardjp@apache.org> ha scritto:
> >
> > > +1 Great idea to make things easier for our users.
> > >
> > > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > > Hi,
> > > >
> > > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > > tries
> > > > to solve some issues we found when enabling Flyway to initialize the
> > > > application database in a compact setup (having runtime with other
> > > modules
> > > > that require persistence in the same app like data-index,
> jobs-service
> > > and
> > > > data-audit). In this setup, a single Flyway instance is executed for
> > the
> > > > whole application during the startup and we are configuring it to
> apply
> > > all
> > > > the SQL scripts (sorted by the version specified in the file name)
> from
> > > all
> > > > those modules and keeping a single index for all the scripts applied
> > > during
> > > > the Flyway execution.
> > > >
> > > > The main problems we identified are related to the versioning of
> those
> > > SQL
> > > > files inside the modules. Flyway gets script information (version,
> > > > description...) from the file name (Ej:
> > > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> > expects
> > > > version to be unique, it doesn't discriminate script source,
> > description.
> > > > Each of our add-on uses its own versions in the SQL scripts (ej:
> > runtime
> > > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > > V1.45.0.4...), in this scenario it is very likely to have collisions
> if
> > > > different modules use the same version in the script name.
> > > > Additionally, it's also possible that for the same reason, DB
> upgrades
> > > > won't work. For example if a module is updated to bring new SQL
> Scripts
> > > to
> > > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > > application
> > > > running a Flyway again won't be able to apply those changes if in a
> > > > previous run another add-on brought a script with a higher version
> (in
> > > this
> > > > case runtime-persistence has v10).
> > > >
> > > > Another issue is that, in this setup, Flyway must be configured to
> know
> > > the
> > > > locations of all the SQL scripts in the applications.properties. In
> > > > general, this SQL files are in different locations, and there are
> > modules
> > > > that provide SQL's for different DB types (h2, pgsql). This can be a
> > bit
> > > > cumbersome to have it properly configured, specially for customers
> that
> > > are
> > > > not familiar with the internals of our code.
> > > >
> > > >
> > > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > > >
> > > > The idea is that instead of using the default Flyway integration
> > provided
> > > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge
> of
> > > > running a separate instance of Flyway for each module that requires
> it
> > > and
> > > > using an independent index for each one them, this way we can ensure
> > that
> > > > there will be no friction between SQL scripts. To make an module
> > > available
> > > > for KIE Flyway, first we must add the following dependency in our
> > module
> > > > pom.xml:
> > > >
> > > > <dependency>
> > > > <groupId>org.kie</groupId>
> > > > <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > > </dependency>
> > > >
> > > > And make sure we include a kie-flyway.properties descriptor located
> at
> > `
> > > > src/main/resources/META-INF` with the following properties:
> > > >
> > > > - module.name: logic name to identify the module.
> > > > - module.locations.<db_type>: this property map is used to setup the
> > > script
> > > > locations per each database supported by our module, (ej:
> > > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > > possible to use the module.locations.default key if we want to use
> some
> > > > fallback SQL's in case the application Database type isn't supported.
> > > This
> > > > locations should refer to a path (or comma-separate paths) unique to
> > the
> > > > module to avoid possible collisions and we should as much as possible
> > > using
> > > > default Flyway locations like db/migrations. You'll notice that I
> moved
> > > the
> > > > current SQL scripts from its location to
> > > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > > >
> > > > You can see an example of this file here (kie-flyway.properties
> > > > <
> > >
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > > >)
> > > > and the new module locations here (locations
> > > > <
> > >
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > > >
> > > > ).
> > > >
> > > >
> > > > This addon supports the following configurations in the
> > > > application.properties file:
> > > >
> > > > - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > > Flyway Add-On on startup, it's disabled by default in prod mode
> > > (Quarkus)
> > > > and fully disabled in Spring-boot.
> > > > - kie.flyway.<module-name>.enabled=true|false # This is used to
> > > disable
> > > > specific module if we need to
> > > >
> > > >
> > > > Some considerations:
> > > >
> > > > - This Add-On is intended to be used for development / test /
> > examples
> > > > and not recommended to use in production
> > > > - As all our Add-Ons will use the default application datasource,
> so
> > > > it's not possible to make it use a different
> > > > - It doesn't cover all the Flyway features, it's only used to run
> > > > migrations to initialize our components.
> > > > - It doesn't replace the Platform Flyway in the kogito
> applications,
> > > the
> > > > user can still use Flyway to load its own scripts without having
> to
> > > care of
> > > > configuring anything from our modules.
> > > >
> > > >
> > > > At this point, for simplicity I only tried to address the DB
> > > Initialization
> > > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > > data-audit &
> > > > user-tasks), but this feature can be also brought to the Data-Index &
> > > Jobs
> > > > Service if we want to, the only reason I didn't do it is to avoid
> doing
> > > > changes that could break the Sonataflow Operator and the examples,
> but
> > > I'll
> > > > happily apply the changes there if there's an agreement on this.
> > > >
> > > > I think that's all... sorry for this long email but I think this
> > > > improvement required some explanations.
> > > >
> > > > Thank you!
> > > >
> > > > Pere
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>
Re: [HEADS UP] KIE Flyway Add-On
Posted by Tibor Zimányi <tz...@apache.org>.
+1
Dňa ut 1. 10. 2024, 8:29 Gabriele Cardosi <ga...@gmail.com>
napísal(a):
> +1
>
> Il giorno lun 30 set 2024 alle ore 23:53 Jason Porter <
> lightguardjp@apache.org> ha scritto:
>
> > +1 Great idea to make things easier for our users.
> >
> > On 2024/09/30 20:32:57 "Pere Fernandez (apache)" wrote:
> > > Hi,
> > >
> > > I recently rised some PR's to introduce a new KIE Flyway Add-On that
> > tries
> > > to solve some issues we found when enabling Flyway to initialize the
> > > application database in a compact setup (having runtime with other
> > modules
> > > that require persistence in the same app like data-index, jobs-service
> > and
> > > data-audit). In this setup, a single Flyway instance is executed for
> the
> > > whole application during the startup and we are configuring it to apply
> > all
> > > the SQL scripts (sorted by the version specified in the file name) from
> > all
> > > those modules and keeping a single index for all the scripts applied
> > during
> > > the Flyway execution.
> > >
> > > The main problems we identified are related to the versioning of those
> > SQL
> > > files inside the modules. Flyway gets script information (version,
> > > description...) from the file name (Ej:
> > > V1.35.0__create_runtime_PostgreSQL.sql -> version is 1.35.0) and
> expects
> > > version to be unique, it doesn't discriminate script source,
> description.
> > > Each of our add-on uses its own versions in the SQL scripts (ej:
> runtime
> > > persistence goes from V1.35.0 to V10.0.1, data-index from V1.32.0 to
> > > V1.45.0.4...), in this scenario it is very likely to have collisions if
> > > different modules use the same version in the script name.
> > > Additionally, it's also possible that for the same reason, DB upgrades
> > > won't work. For example if a module is updated to bring new SQL Scripts
> > to
> > > upgrade the DB (let's say data-index bringing V1.5.0) the same
> > application
> > > running a Flyway again won't be able to apply those changes if in a
> > > previous run another add-on brought a script with a higher version (in
> > this
> > > case runtime-persistence has v10).
> > >
> > > Another issue is that, in this setup, Flyway must be configured to know
> > the
> > > locations of all the SQL scripts in the applications.properties. In
> > > general, this SQL files are in different locations, and there are
> modules
> > > that provide SQL's for different DB types (h2, pgsql). This can be a
> bit
> > > cumbersome to have it properly configured, specially for customers that
> > are
> > > not familiar with the internals of our code.
> > >
> > >
> > > So, in order to help here, we are adding this Kie Flyway Add-On.
> > >
> > > The idea is that instead of using the default Flyway integration
> provided
> > > by the Platform (Quarkus/Spring-Boot), this Add-on will be in charge of
> > > running a separate instance of Flyway for each module that requires it
> > and
> > > using an independent index for each one them, this way we can ensure
> that
> > > there will be no friction between SQL scripts. To make an module
> > available
> > > for KIE Flyway, first we must add the following dependency in our
> module
> > > pom.xml:
> > >
> > > <dependency>
> > > <groupId>org.kie</groupId>
> > > <artifactId>kie-addons-<quarkus|springboot>-flyway</artifactId>
> > > </dependency>
> > >
> > > And make sure we include a kie-flyway.properties descriptor located at
> `
> > > src/main/resources/META-INF` with the following properties:
> > >
> > > - module.name: logic name to identify the module.
> > > - module.locations.<db_type>: this property map is used to setup the
> > script
> > > locations per each database supported by our module, (ej:
> > > module.locations.postgresql=classpath:<path-to-my-sql-files>). It's
> > > possible to use the module.locations.default key if we want to use some
> > > fallback SQL's in case the application Database type isn't supported.
> > This
> > > locations should refer to a path (or comma-separate paths) unique to
> the
> > > module to avoid possible collisions and we should as much as possible
> > using
> > > default Flyway locations like db/migrations. You'll notice that I moved
> > the
> > > current SQL scripts from its location to
> > > src/main/resources/kie-flyway/db/<module-name>/<db-type>
> > >
> > > You can see an example of this file here (kie-flyway.properties
> > > <
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/blob/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/META-INF/kie-flyway.properties
> > >)
> > > and the new module locations here (locations
> > > <
> >
> https://github.com/apache/incubator-kie-kogito-runtimes/tree/5037c9de5608fde136fef274effd3045c91c162f/addons/common/persistence/jdbc/src/main/resources/kie-flyway/db/persistence-jdbc
> > >
> > > ).
> > >
> > >
> > > This addon supports the following configurations in the
> > > application.properties file:
> > >
> > > - kie.flyway.enabled=true|false # Enables the execution of the KIE
> > > Flyway Add-On on startup, it's disabled by default in prod mode
> > (Quarkus)
> > > and fully disabled in Spring-boot.
> > > - kie.flyway.<module-name>.enabled=true|false # This is used to
> > disable
> > > specific module if we need to
> > >
> > >
> > > Some considerations:
> > >
> > > - This Add-On is intended to be used for development / test /
> examples
> > > and not recommended to use in production
> > > - As all our Add-Ons will use the default application datasource, so
> > > it's not possible to make it use a different
> > > - It doesn't cover all the Flyway features, it's only used to run
> > > migrations to initialize our components.
> > > - It doesn't replace the Platform Flyway in the kogito applications,
> > the
> > > user can still use Flyway to load its own scripts without having to
> > care of
> > > configuring anything from our modules.
> > >
> > >
> > > At this point, for simplicity I only tried to address the DB
> > Initialization
> > > of our Add-Ons (runtime persistence, data-index, jobs-service,
> > data-audit &
> > > user-tasks), but this feature can be also brought to the Data-Index &
> > Jobs
> > > Service if we want to, the only reason I didn't do it is to avoid doing
> > > changes that could break the Sonataflow Operator and the examples, but
> > I'll
> > > happily apply the changes there if there's an agreement on this.
> > >
> > > I think that's all... sorry for this long email but I think this
> > > improvement required some explanations.
> > >
> > > Thank you!
> > >
> > > Pere
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>