You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@isis.apache.org by Kambiz Darabi <da...@m-creations.com> on 2016/06/20 14:27:53 UTC
Integrating Flyway for database migrations
Hi,
in our non-Isis projects, we use FlyWay [1] for DB migrations and I
would like to integrate it into our Isis workflow. The simplest path to
do so would be a DomainService with a PostConstruct annotated init
method:
@PostConstruct
public void init(final Map<String, String> properties) {
Flyway flyway = new Flyway();
// Point it to the database
String jdbcUrl = properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionURL");
String user = properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionUserName");
String password = properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionPassword");
flyway.setDataSource(jdbcUrl, user, password);
flyway.setLocations("classpath:db/migrations");
// Start the migration
flyway.migrate();
}
but this isn't a viable solution, as IsisSessionFactoryBuilder's
buildSessionFactory() method initialises the DataNucleus (DN)
PersistenceSessionFactory before the services are constructed [2].
So DN has already found the mismatch between the JDO annotations and the
database before we enter the init method of our DB migration
bootstrap/seed service.
I could contribute a patch, if someone could hint on the preferred way
of implementing the functionality.
Thank you
Kambiz
[1] https://flywaydb.org/
[2] https://github.com/apache/isis/blob/master/core/runtime/src/main/java/org/apache/isis/core/runtime/system/session/IsisSessionFactoryBuilder.java#L184
Re: Integrating Flyway for database migrations
Posted by Jeroen van der Wal <je...@stromboli.it>.
Hi Oscar,
We're dropping the constraints because it makes it easier to handle
migrations and were then 100% sure that no retired constraint stays behind.
I've not seen the error that you are referring to. We're also running some
other Apache Isis applications on PostgreSQL so it might be something else
that is causing your issue.
Cheers,
Jeroen
On 22 June 2016 at 10:07, Óscar Bou - GOVERTIS <o....@govertis.com> wrote:
> Hi, Jeroen.
>
> Nice to know you drop the constraints when upgrading.
>
> I’ve also noticed that they get duplicated. Seems DN does not detect
> they’re already created and re-create them.
> In my case the db is PostgreSQL. As you’re using MS SQL Server in Estatio,
> seems it can be a generic issue.
>
> It’s strange DN does not have addressed it.
>
> Please, confirm me if that’s the issue causing you to delete the
> constraints and I would create a issue in the DN ticketing system.
>
>
> Thanks,
>
> Oscar
>
>
>
> El 21 jun 2016, a las 23:56, Jeroen van der Wal <je...@stromboli.it>
> escribió:
>
> Hi Kambiz,
>
> There's currently not a nice hook that I can think of to execute Flyway
> migrations. I would create a separate "upgrade" mode to start Isis that
> bootstraps with an in-memory db and allows you to do the Flyway stuff. But
> Dan probably has other ideas ;-)
>
> I've looked into Flyway for exactly the same purpose but was not really
> enthousiast about it. What I disliked the most is that you have to maintain
> every single db change in scripts. For me, the domain model is the source
> and persistence should be derived from that. And Datanucleus does an
> excellent job in creating all database artifacts so I want to keep
> leveraging that.
>
> What we currently do (manually) is roughly this:
> 1. stop Isis;
> 2. drop all db constraints;
> 3. apply db upgrade script (for the changes that cannot be handled by
> Datanucleus);
> 4. start Isis;
> 5. execute upgrade service (for programmatic changes).
>
> We are also trying to crack the nut on how to automate this but encounter a
> few hurdles and I am not sure if Flyway can tackle those:
> - we have applications that consist of multiple modules, each with its own
> db schema and that change independently and the application should
> orchestrate the right order of upgrading;
> - a lot of times data is migrated, even between schemas and we sometimes
> use temporary views to do a pre and post check;
>
> Any ideas are welcome.
>
> Cheers,
>
> Jeroen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 20 June 2016 at 16:27, Kambiz Darabi <da...@m-creations.com> wrote:
>
> Hi,
>
> in our non-Isis projects, we use FlyWay [1] for DB migrations and I
> would like to integrate it into our Isis workflow. The simplest path to
> do so would be a DomainService with a PostConstruct annotated init
> method:
>
> @PostConstruct
> public void init(final Map<String, String> properties) {
> Flyway flyway = new Flyway();
>
> // Point it to the database
> String jdbcUrl =
>
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionURL");
> String user =
>
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionUserName");
> String password =
>
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionPassword");
>
> flyway.setDataSource(jdbcUrl, user, password);
> flyway.setLocations("classpath:db/migrations");
> // Start the migration
> flyway.migrate();
> }
>
>
> but this isn't a viable solution, as IsisSessionFactoryBuilder's
> buildSessionFactory() method initialises the DataNucleus (DN)
> PersistenceSessionFactory before the services are constructed [2].
>
> So DN has already found the mismatch between the JDO annotations and the
> database before we enter the init method of our DB migration
> bootstrap/seed service.
>
> I could contribute a patch, if someone could hint on the preferred way
> of implementing the functionality.
>
> Thank you
>
>
> Kambiz
>
>
> [1] https://flywaydb.org/
>
> [2]
>
> https://github.com/apache/isis/blob/master/core/runtime/src/main/java/org/apache/isis/core/runtime/system/session/IsisSessionFactoryBuilder.java#L184
>
>
>
>
> Óscar Bou Bou
> Socio - IT & GRC Management Services Director
> m: +34 620 267 520
> s: <http://www.govertis.com>www.govertis.com e: o.bou@govertis.com
>
> LinkedIn: https://www.linkedin.com/in/oscarbou
> Twitter: @oscarbou <https://twitter.com/oscarbou>
>
>
>
> Este mensaje y los ficheros anexos son confidenciales. Los mismos
> contienen información reservada que no puede ser difundida. Si usted ha
> recibido este correo por error, tenga la amabilidad de eliminarlo de su
> sistema y avisar al remitente mediante reenvío a su dirección electrónica;
> no deberá copiar el mensaje ni divulgar su contenido a ninguna persona.
>
> Su dirección de correo electrónico junto a sus datos personales constan en
> un fichero titularidad de GOVERTIS ADVISORY SERVICES, S.L. cuya finalidad
> es la de mantener el contacto con Ud. Si quiere saber de qué información
> disponemos de Ud., modificarla, y en su caso, cancelarla, puede hacerlo
> enviando un escrito al efecto, acompañado de una fotocopia de su D.N.I. a
> la siguiente dirección: GOVERTIS ADVISORY SERVICES, S.L. Avda Cortes
> Valencianas, 58 – 8º - 6ª. 46015 - Valencia, y Paseo de la Castellana,
> 153, 28045 - MADRID. Asimismo, es su responsabilidad comprobar que este
> mensaje o sus archivos adjuntos no contengan virus informáticos, y en caso
> que los tuvieran eliminarlos.
>
>
Re: Integrating Flyway for database migrations
Posted by Óscar Bou - GOVERTIS <o....@govertis.com>.
Hi, Jeroen.
Nice to know you drop the constraints when upgrading.
I’ve also noticed that they get duplicated. Seems DN does not detect they’re already created and re-create them.
In my case the db is PostgreSQL. As you’re using MS SQL Server in Estatio, seems it can be a generic issue.
It’s strange DN does not have addressed it.
Please, confirm me if that’s the issue causing you to delete the constraints and I would create a issue in the DN ticketing system.
Thanks,
Oscar
> El 21 jun 2016, a las 23:56, Jeroen van der Wal <je...@stromboli.it> escribió:
>
> Hi Kambiz,
>
> There's currently not a nice hook that I can think of to execute Flyway
> migrations. I would create a separate "upgrade" mode to start Isis that
> bootstraps with an in-memory db and allows you to do the Flyway stuff. But
> Dan probably has other ideas ;-)
>
> I've looked into Flyway for exactly the same purpose but was not really
> enthousiast about it. What I disliked the most is that you have to maintain
> every single db change in scripts. For me, the domain model is the source
> and persistence should be derived from that. And Datanucleus does an
> excellent job in creating all database artifacts so I want to keep
> leveraging that.
>
> What we currently do (manually) is roughly this:
> 1. stop Isis;
> 2. drop all db constraints;
> 3. apply db upgrade script (for the changes that cannot be handled by
> Datanucleus);
> 4. start Isis;
> 5. execute upgrade service (for programmatic changes).
>
> We are also trying to crack the nut on how to automate this but encounter a
> few hurdles and I am not sure if Flyway can tackle those:
> - we have applications that consist of multiple modules, each with its own
> db schema and that change independently and the application should
> orchestrate the right order of upgrading;
> - a lot of times data is migrated, even between schemas and we sometimes
> use temporary views to do a pre and post check;
>
> Any ideas are welcome.
>
> Cheers,
>
> Jeroen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 20 June 2016 at 16:27, Kambiz Darabi <da...@m-creations.com> wrote:
>
>> Hi,
>>
>> in our non-Isis projects, we use FlyWay [1] for DB migrations and I
>> would like to integrate it into our Isis workflow. The simplest path to
>> do so would be a DomainService with a PostConstruct annotated init
>> method:
>>
>> @PostConstruct
>> public void init(final Map<String, String> properties) {
>> Flyway flyway = new Flyway();
>>
>> // Point it to the database
>> String jdbcUrl =
>> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionURL");
>> String user =
>> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionUserName");
>> String password =
>> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionPassword");
>>
>> flyway.setDataSource(jdbcUrl, user, password);
>> flyway.setLocations("classpath:db/migrations");
>> // Start the migration
>> flyway.migrate();
>> }
>>
>>
>> but this isn't a viable solution, as IsisSessionFactoryBuilder's
>> buildSessionFactory() method initialises the DataNucleus (DN)
>> PersistenceSessionFactory before the services are constructed [2].
>>
>> So DN has already found the mismatch between the JDO annotations and the
>> database before we enter the init method of our DB migration
>> bootstrap/seed service.
>>
>> I could contribute a patch, if someone could hint on the preferred way
>> of implementing the functionality.
>>
>> Thank you
>>
>>
>> Kambiz
>>
>>
>> [1] https://flywaydb.org/
>>
>> [2]
>> https://github.com/apache/isis/blob/master/core/runtime/src/main/java/org/apache/isis/core/runtime/system/session/IsisSessionFactoryBuilder.java#L184
>>
>>
Óscar Bou Bou
Socio - IT & GRC Management Services Director
m: +34 620 267 520
s: <http://www.govertis.com/>www.govertis.com <http://www.govertis.com/> e: o.bou@govertis.com <ma...@govertis.com>
LinkedIn: https://www.linkedin.com/in/oscarbou <https://www.linkedin.com/in/oscarbou>
Twitter: @oscarbou <https://twitter.com/oscarbou>
Este mensaje y los ficheros anexos son confidenciales. Los mismos contienen información reservada que no puede ser difundida. Si usted ha recibido este correo por error, tenga la amabilidad de eliminarlo de su sistema y avisar al remitente mediante reenvío a su dirección electrónica; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona.
Su dirección de correo electrónico junto a sus datos personales constan en un fichero titularidad de GOVERTIS ADVISORY SERVICES, S.L. cuya finalidad es la de mantener el contacto con Ud. Si quiere saber de qué información disponemos de Ud., modificarla, y en su caso, cancelarla, puede hacerlo enviando un escrito al efecto, acompañado de una fotocopia de su D.N.I. a la siguiente dirección: GOVERTIS ADVISORY SERVICES, S.L. Avda Cortes Valencianas, 58 – 8º - 6ª. 46015 - Valencia, y Paseo de la Castellana, 153, 28045 - MADRID. Asimismo, es su responsabilidad comprobar que este mensaje o sus archivos adjuntos no contengan virus informáticos, y en caso que los tuvieran eliminarlos.
Re: Integrating Flyway for database migrations
Posted by Jeroen van der Wal <je...@stromboli.it>.
Hi Kambiz,
There's currently not a nice hook that I can think of to execute Flyway
migrations. I would create a separate "upgrade" mode to start Isis that
bootstraps with an in-memory db and allows you to do the Flyway stuff. But
Dan probably has other ideas ;-)
I've looked into Flyway for exactly the same purpose but was not really
enthousiast about it. What I disliked the most is that you have to maintain
every single db change in scripts. For me, the domain model is the source
and persistence should be derived from that. And Datanucleus does an
excellent job in creating all database artifacts so I want to keep
leveraging that.
What we currently do (manually) is roughly this:
1. stop Isis;
2. drop all db constraints;
3. apply db upgrade script (for the changes that cannot be handled by
Datanucleus);
4. start Isis;
5. execute upgrade service (for programmatic changes).
We are also trying to crack the nut on how to automate this but encounter a
few hurdles and I am not sure if Flyway can tackle those:
- we have applications that consist of multiple modules, each with its own
db schema and that change independently and the application should
orchestrate the right order of upgrading;
- a lot of times data is migrated, even between schemas and we sometimes
use temporary views to do a pre and post check;
Any ideas are welcome.
Cheers,
Jeroen
On 20 June 2016 at 16:27, Kambiz Darabi <da...@m-creations.com> wrote:
> Hi,
>
> in our non-Isis projects, we use FlyWay [1] for DB migrations and I
> would like to integrate it into our Isis workflow. The simplest path to
> do so would be a DomainService with a PostConstruct annotated init
> method:
>
> @PostConstruct
> public void init(final Map<String, String> properties) {
> Flyway flyway = new Flyway();
>
> // Point it to the database
> String jdbcUrl =
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionURL");
> String user =
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionUserName");
> String password =
> properties.get("isis.persistor.datanucleus.impl.javax.jdo.option.ConnectionPassword");
>
> flyway.setDataSource(jdbcUrl, user, password);
> flyway.setLocations("classpath:db/migrations");
> // Start the migration
> flyway.migrate();
> }
>
>
> but this isn't a viable solution, as IsisSessionFactoryBuilder's
> buildSessionFactory() method initialises the DataNucleus (DN)
> PersistenceSessionFactory before the services are constructed [2].
>
> So DN has already found the mismatch between the JDO annotations and the
> database before we enter the init method of our DB migration
> bootstrap/seed service.
>
> I could contribute a patch, if someone could hint on the preferred way
> of implementing the functionality.
>
> Thank you
>
>
> Kambiz
>
>
> [1] https://flywaydb.org/
>
> [2]
> https://github.com/apache/isis/blob/master/core/runtime/src/main/java/org/apache/isis/core/runtime/system/session/IsisSessionFactoryBuilder.java#L184
>
>