Upgrade from 1.1.X to 1.2.X - how to configure internal storage?

[Guido Wimmel <gu...@gmx.net>]

Hi,

I'm currently evaluating the Syncope upgrade procedure from 1.1.X to
1.2.X (see the description in the Wiki[1]).

I'm not sure what is the recommended way to deal with the internal storage
(src/main/resources/content.xml).
The Wiki page says at first that one should "configure the internal storage
as done for the former 1.1.X project", then "change content.xml leaving
only new Entitlements".

Should one have a specific version of content.xml that is only relevant for
the upgrade? (which then cannot be used to set up a new instance from an
empty database)
Should everything be commented out but the new entitlements, or should one
only comment out the entitlements that already exist in the database
and leave other declarations in (such as policies)? If I comment out everything
but the new entitlements, I get an error message about a missing
notificationJob.cronExpression  (NullPointerException) on initialisation
(in JobInstanceLoader).
If I leave other declarations (such as config settings) in, won't they
overwrite the existing configuration in my 1.1.X database?
There seems to be a mechanism in content upgrader that explicitly doesn't
migrate some config settings (such as notificationJob.cronExpression) -
must these then be manually set again to their previous values in the
1.1.X database?

Cheers,
  Guido

[1] https://cwiki.apache.org/confluence/display/SYNCOPE/Upgrade+from+1.1.X+to+1.2.X

Re: Aw: Re: Upgrade from 1.1.X to 1.2.X - how to configure internal storage?

[Andrea Patricelli <an...@tirasa.net>]

----- Messaggio originale -----
Da: "Guido Wimmel" <gu...@gmx.net>
A: user@syncope.apache.org
Inviato: Giovedì, 20 novembre 2014 8:38:30
Oggetto: Aw: Re: Upgrade from 1.1.X to 1.2.X - how to configure internal storage?

Hi Andrea,

thanks for the reply.

> > The Wiki page says at first that one should "configure the internal storage
> > as done for the former 1.1.X project", then "change content.xml leaving
> > only new Entitlements".
> >
> > Should one have a specific version of content.xml that is only relevant for
> > the upgrade? (which then cannot be used to set up a new instance from an
> > empty database)
> No, there isn't a custom upgrade-purpose content.xml.
> You have to use "basic" content.xml, that is the one you have when you 
> create a new Apache Syncope project, as explained in the wiki.

>Ok, but somehow I do need to modify this into a migration-specific
>content.xml (e.g. with the existing entitlements commented out, see below,
>which then wouldn't work anymore with an empty database),
>and then change it back after migration, right?

Yes. The content.xml you modify is not correct for a clean, "ex-novo" installation, You should modify it "rollbacking" comments on Entitlements if you'll need it in the future.

> > If I leave other declarations (such as config settings) in, won't they
> > overwrite the existing configuration in my 1.1.X database?
> > There seems to be a mechanism in content upgrader that explicitly doesn't
> > migrate some config settings (such as notificationJob.cronExpression) -
> > must these then be manually set again to their previous values in the
> > 1.1.X database?
> This point you noticed is interesting... The answer is yes, new 
> content.xml conf have priority and if in the "old" conf 1.1.X table 
> contains the same key with a value it will be written the "new" value 
> get from the content.xml mentioned above, especially the old value is 
> escaped and if the conf (that the upgrader finds) is in the list of new 
> confs, value isn't written to database.
> And so yes you have to manually set the value if you want old one.
> Maybe this section has to be re-thinked and improved, I will open an 
> issue about this problem.

>That would be very helpful - especially to have information which part of the
>configuration is migrated from the 1.1.X database and for which part of the configuration
>the settings in the 1.1.X database are ignored and overwritten with the values in
>content.xml (thus have to be migrated manually).

My idea was to migrate all "old" 1.1.X configuration, i.e. all conf objects that don't match with the objects in a specific list (like a black list:) ). This list is, for now, an enum in the code called SyncopeDefParams (that, moreover, must be updated when added new default configuration parameter).
Then, with resolution of [2], configuration existing in 1.2.X and 1.1.X could be updated with old values got from 1.1.X database (because we have all of these values during upgrade), so nothing must be made manually but the content.xml modification. 

>About the other settings in content.xml I'm still not sure how this is supposed to work exactly.
>I understood content.xml as default content set up on database schema creation on an empty database,
>so for me it also contains e.g. predefined user schema attributes, policy declarations and
>notifications. On the upgrade, the ContentLoader tries to add these as well (and usually fails
>because of a duplicate key). However, in the general case one couldn't rely on the ContentLoader failing -
>maybe a user schema that was there initially was deleted via the GUI, then it's added again in the migration...

Maybe I haven't understood well the question, but the content.xml we refer in the wiki is basic content.xml, containing only SyncopeConf, Notification, Notification_events, conf objects and Entitlements, nothing else.
This situation so can occur only for Notification, not for user schema or policies because they aren't in the default content.xml (that is what you have when you create new project from archetype).
So maybe this was a misleading point, content.xml I referred was archetype basic content.xml, I don't think that upgrade will work with other contents containing other objects.

>Aren't policies, user schemata and notifications migrated from the 1.1.X database?
>Wouldn't it be better to leave only the declarations in content.xml that are absolutely necessary for the
>migration to work and take as much as possible from the 1.1.X database?

Yes all of these objects are migrated from 1.1.X database (and not initialized by content.xml), in fact the content.xml we was refering was the minimal one, moreover, thinking about what you pointed out above, I think that also notification can be commented and other sections are mandatory.
In case of commenting Notification I think that nothing appens, on the contrary it will be resolved the issue of re-entering deleted Notifications. But this thing must be tested.
Moreover in the wiki is written that use must "Save Notification "about" and "recipients", upgrade sets default values for these fileds due to radical change to Notification object from 1.1.X to 1.2.X". Notification handling was a bit tricky.

>Cheers,
>  Guido

Available for other clarifications.

Cheers,
Andrea

> > [1] https://cwiki.apache.org/confluence/display/SYNCOPE/Upgrade+from+1.1.X+to+1.2.X
> I'm here available for all other questions.

[2]https://issues.apache.org/jira/browse/SYNCOPE-609

Aw: Re: Upgrade from 1.1.X to 1.2.X - how to configure internal storage?

[Guido Wimmel <gu...@gmx.net>]

Hi Andrea,

thanks for the reply.

> > The Wiki page says at first that one should "configure the internal storage
> > as done for the former 1.1.X project", then "change content.xml leaving
> > only new Entitlements".
> >
> > Should one have a specific version of content.xml that is only relevant for
> > the upgrade? (which then cannot be used to set up a new instance from an
> > empty database)
> No, there isn't a custom upgrade-purpose content.xml.
> You have to use "basic" content.xml, that is the one you have when you 
> create a new Apache Syncope project, as explained in the wiki.

Ok, but somehow I do need to modify this into a migration-specific
content.xml (e.g. with the existing entitlements commented out, see below,
which then wouldn't work anymore with an empty database),
and then change it back after migration, right?


> > If I leave other declarations (such as config settings) in, won't they
> > overwrite the existing configuration in my 1.1.X database?
> > There seems to be a mechanism in content upgrader that explicitly doesn't
> > migrate some config settings (such as notificationJob.cronExpression) -
> > must these then be manually set again to their previous values in the
> > 1.1.X database?
> This point you noticed is interesting... The answer is yes, new 
> content.xml conf have priority and if in the "old" conf 1.1.X table 
> contains the same key with a value it will be written the "new" value 
> get from the content.xml mentioned above, especially the old value is 
> escaped and if the conf (that the upgrader finds) is in the list of new 
> confs, value isn't written to database.
> And so yes you have to manually set the value if you want old one.
> Maybe this section has to be re-thinked and improved, I will open an 
> issue about this problem.

That would be very helpful - especially to have information which part of the
configuration is migrated from the 1.1.X database and for which part of the configuration
the settings in the 1.1.X database are ignored and overwritten with the values in
content.xml (thus have to be migrated manually).

About the other settings in content.xml I'm still not sure how this is supposed to work exactly.
I understood content.xml as default content set up on database schema creation on an empty database,
so for me it also contains e.g. predefined user schema attributes, policy declarations and
notifications. On the upgrade, the ContentLoader tries to add these as well (and usually fails
because of a duplicate key). However, in the general case one couldn't rely on the ContentLoader failing -
maybe a user schema that was there initially was deleted via the GUI, then it's added again in the migration...

Aren't policies, user schemata and notifications migrated from the 1.1.X database?
Wouldn't it be better to leave only the declarations in content.xml that are absolutely necessary for the
migration to work and take as much as possible from the 1.1.X database?

Cheers,
  Guido


> > [1] https://cwiki.apache.org/confluence/display/SYNCOPE/Upgrade+from+1.1.X+to+1.2.X
> I'm here available for all other questions.

Re: Upgrade from 1.1.X to 1.2.X - how to configure internal storage?

[andrea patricelli <an...@tirasa.net>]

Il 19/11/2014 14:17, Guido Wimmel ha scritto:
> Hi,
Hi Guido,
>
> I'm currently evaluating the Syncope upgrade procedure from 1.1.X to
> 1.2.X (see the description in the Wiki[1]).
>
> I'm not sure what is the recommended way to deal with the internal storage
> (src/main/resources/content.xml).
> The Wiki page says at first that one should "configure the internal storage
> as done for the former 1.1.X project", then "change content.xml leaving
> only new Entitlements".
>
> Should one have a specific version of content.xml that is only relevant for
> the upgrade? (which then cannot be used to set up a new instance from an
> empty database)
No, there isn't a custom upgrade-purpose content.xml.
You have to use "basic" content.xml, that is the one you have when you 
create a new Apache Syncope project, as explained in the wiki.
> Should everything be commented out but the new entitlements, or should one
> only comment out the entitlements that already exist in the database
> and leave other declarations in (such as policies)? If I comment out everything
> but the new entitlements, I get an error message about a missing
> notificationJob.cronExpression  (NullPointerException) on initialisation
> (in JobInstanceLoader).
No you have to touch only Entitlements section, maybe in the wiki it was 
explained in a misleading way. All other sections must remain the same 
(untouched).
It's correct that you have a NPE because, if you comment out other 
sections such as policies, conf, etc, there aren't useful information.
I changed the wiki to be more explanatory.
> If I leave other declarations (such as config settings) in, won't they
> overwrite the existing configuration in my 1.1.X database?
> There seems to be a mechanism in content upgrader that explicitly doesn't
> migrate some config settings (such as notificationJob.cronExpression) -
> must these then be manually set again to their previous values in the
> 1.1.X database?
This point you noticed is interesting... The answer is yes, new 
content.xml conf have priority and if in the "old" conf 1.1.X table 
contains the same key with a value it will be written the "new" value 
get from the content.xml mentioned above, especially the old value is 
escaped and if the conf (that the upgrader finds) is in the list of new 
confs, value isn't written to database.
And so yes you have to manually set the value if you want old one.
Maybe this section has to be re-thinked and improved, I will open an 
issue about this problem.
>
> Cheers,
>    Guido
>
> [1] https://cwiki.apache.org/confluence/display/SYNCOPE/Upgrade+from+1.1.X+to+1.2.X
I'm here available for all other questions.

Best regards,
Andrea

-- 
Dott. Andrea Patricelli
Tel +393204524292

Junior Engineer @ Tirasa S.r.l.
Viale D'Annunzio 267 - 65127 Pescara
Tel +390859116307 / FAX +39 0859111173
http://www.tirasa.net