You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@syncope.apache.org by Mikael Ekblom <mi...@arcada.fi> on 2017/04/20 08:03:46 UTC
Question regarding time formats
Hi,
We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.
I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:
09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93 Method: handleAttribute
java.lang.NullPointerException: null
First question:
Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter?
In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.
Second question:
When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope. Not even with pull mode set to "Full reconciliation" funny enough.
What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?
Regards,
Mikael
Mikael Ekblom
Systemutvecklare/System developer
Arcada, IT
Jan-Magnus Janssons plats 1,
FIN-00560 Helsingfors,
Finland
TFn: +358 207 699 467
Mobil: +358 207 699 467
Re: Question regarding time formats
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 21/04/2017 11:32, Mikael Ekblom wrote:
>
> Hi,
>
> Sure, I can file an issue on Jira. As for now, we do get the dates
> synchronized in the correct manner from the right source.
>
Don't mind, I went ahead and created
https://issues.apache.org/jira/browse/SYNCOPE-1070
Regards.
> I was reading about the DB connector and I was sure that it stated
> that it supported delete as optional:
>
> Authenticate (*/Optional/*) Specify the password column configuration
> property.
>
> �Create
>
> �Delete
>
> �Update
>
> �Search
>
> �Schema
>
> �Test
>
> �Sync (*/Optional/*) Specify the Change Log Column configuration property.
>
> I thought that optional was a notation for the fact that it could be
> implemented by having the change log column associated with the
> resource. I\u2019ll look more into the groovy-scripts. We have had some
> parsing problems with those, but that might be due to some file
> encoding issue. Therefore we collected the base master data together
> from various sources and thought of following the \u201ckeep it simple
> stupid\u201d policy with the db-connector\u2026J
>
> Regards,
>
> Mikael
>
> *From:*Francesco Chicchiricc� [mailto:ilgrosso@apache.org]
> *Sent:* perjantai 21. huhtikuuta 2017 11.52
> *To:* user@syncope.apache.org
> *Subject:* Re: Question regarding time formats
>
> Hi Mikael,
> see my replies below.
>
> Regards.
>
> On 20/04/2017 10:03, Mikael Ekblom wrote:
>
> Hi,
>
> We have now implemented a pullaction, that is able to
> automatically generate usernames and passwords from a mssql
> database source, that is simulating our HR-system. So far so good.
>
>
> Cool.
>
>
> I can also see, that syncope is able to fetch start and end dates
> automatically from the db-source(date stated as datetime column),
> but, when we try to sync it the other way, we get an exception:
>
> 09:40:52.331 ERROR Error parsing value {0} of attribute
> Slutdatum:93 Method: handleAttribute
>
> java.lang.NullPointerException: null
>
> *First question:*
>
> Is it now so, that syncope is able to transform and understand a
> datetime format quite easily into a correct java.date format when
> it pulls this information from an external source, but the
> conversion the other way round is troublesome when
> pushing/propagating data towards the external resource? Do I need
> to transform it first before the propagation? Maybe a SimpleDate
> formatter?
>
>
> When Syncope fetches data from the db-source, we are doing pull.
> Consider that such data is arriving in a format according to ConnId
> specifications; FYI, ConnId supports the following types:
>
> https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212
>
> where, as you can see, there is no Date.
>
> This means that the date values arrive as text, and are parsed by
> Syncope according to the format specified for the mapped internal schema.
>
> At the moment, however, during propagation the eventual conversion
> pattern is simply not taken into account:
>
> https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263
>
> There should be room for an improvement here: would you mind to file
> an issue on JIRA about propagating Date values as formatted strings?
>
>
> In that sense, it is better that it fails the other way around,
> because we are to fetch the values from the HR and they should not
> be managed from the syncope environment.
>
>
> If such values are not to be used anywhere else in Syncope, then
> simply define the internal schema as String rather than Date.
>
>
> *Second question:*
>
> When a connector has the SEARCH and SYNC operations associated
> with itself: how is the logic supposed to work if you do have the
> change log column set and deletes a tuple in the external
> database? Pull mode is set to incremental. On our end here, even
> though we delete the tuple in the mssql hr simulation database,
> the associated account is not deleted within syncope. Not even
> with pull mode set to \u201cFull reconciliation\u201d funny enough.
>
> What I can see, the pull action handler is not logging anything if
> the pull mode is not set to full reconciliation either. Syncope
> and the db do have the associated key values set correctly etc.
> The change log column is of a datetime format also. Does this has
> something to do with it?
>
>
> You need to consider that everything related to SYNC mostly depends on
> how the actual connector implements such operation.
> The db-table connector is possibly one of the simplest out there;
> among its limitation, it is also not capable to feed Syncope with
> information about deleted users.
>
> I would suggest to use the more flexible Scripted SQL for the job;
> you'll need, however, to provide your own Groovy scripts.
>
> You might want to look at
>
> https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql
>
> as starting point; FYI, look how one of such sample scripts can
> generate DELETE events to send back to Syncope:
>
> https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78
>
> by simply considering an additional 'deleted' column.
>
> --
> Francesco Chicchiricc�
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
> Member at The Apache Software Foundation
> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> http://home.apache.org/~ilgrosso/ <http://home.apache.org/%7Eilgrosso/>
--
Francesco Chicchiricc�
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
RE: Question regarding time formats
Posted by Mikael Ekblom <mi...@arcada.fi>.
Hi,
Sure, I can file an issue on Jira. As for now, we do get the dates synchronized in the correct manner from the right source.
I was reading about the DB connector and I was sure that it stated that it supported delete as optional:
Authenticate (Optional) Specify the password column configuration property.
· Create
· Delete
· Update
· Search
· Schema
· Test
· Sync (Optional) Specify the Change Log Column configuration property.
I thought that optional was a notation for the fact that it could be implemented by having the change log column associated with the resource. I’ll look more into the groovy-scripts. We have had some parsing problems with those, but that might be due to some file encoding issue. Therefore we collected the base master data together from various sources and thought of following the “keep it simple stupid” policy with the db-connector…☺
Regards,
Mikael
From: Francesco Chicchiriccò [mailto:ilgrosso@apache.org]
Sent: perjantai 21. huhtikuuta 2017 11.52
To: user@syncope.apache.org
Subject: Re: Question regarding time formats
Hi Mikael,
see my replies below.
Regards.
On 20/04/2017 10:03, Mikael Ekblom wrote:
Hi,
We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.
Cool.
I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:
09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93 Method: handleAttribute
java.lang.NullPointerException: null
First question:
Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter?
When Syncope fetches data from the db-source, we are doing pull.
Consider that such data is arriving in a format according to ConnId specifications; FYI, ConnId supports the following types:
https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212
where, as you can see, there is no Date.
This means that the date values arrive as text, and are parsed by Syncope according to the format specified for the mapped internal schema.
At the moment, however, during propagation the eventual conversion pattern is simply not taken into account:
https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263
There should be room for an improvement here: would you mind to file an issue on JIRA about propagating Date values as formatted strings?
In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.
If such values are not to be used anywhere else in Syncope, then simply define the internal schema as String rather than Date.
Second question:
When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope. Not even with pull mode set to “Full reconciliation” funny enough.
What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?
You need to consider that everything related to SYNC mostly depends on how the actual connector implements such operation.
The db-table connector is possibly one of the simplest out there; among its limitation, it is also not capable to feed Syncope with information about deleted users.
I would suggest to use the more flexible Scripted SQL for the job; you'll need, however, to provide your own Groovy scripts.
You might want to look at
https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql
as starting point; FYI, look how one of such sample scripts can generate DELETE events to send back to Syncope:
https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78
by simply considering an additional 'deleted' column.
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Question regarding time formats
Posted by Francesco Chicchiriccò <il...@apache.org>.
Hi Mikael,
see my replies below.
Regards.
On 20/04/2017 10:03, Mikael Ekblom wrote:
>
> Hi,
>
> We have now implemented a pullaction, that is able to automatically
> generate usernames and passwords from a mssql database source, that is
> simulating our HR-system. So far so good.
>
Cool.
> I can also see, that syncope is able to fetch start and end dates
> automatically from the db-source(date stated as datetime column), but,
> when we try to sync it the other way, we get an exception:
>
> 09:40:52.331 ERROR Error parsing value {0} of attribute
> Slutdatum:93 Method: handleAttribute
>
> java.lang.NullPointerException: null
>
> *First question:*
>
> Is it now so, that syncope is able to transform and understand a
> datetime format quite easily into a correct java.date format when it
> pulls this information from an external source, but the conversion the
> other way round is troublesome when pushing/propagating data towards
> the external resource? Do I need to transform it first before the
> propagation? Maybe a SimpleDate formatter?
>
When Syncope fetches data from the db-source, we are doing pull.
Consider that such data is arriving in a format according to ConnId
specifications; FYI, ConnId supports the following types:
https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212
where, as you can see, there is no Date.
This means that the date values arrive as text, and are parsed by
Syncope according to the format specified for the mapped internal schema.
At the moment, however, during propagation the eventual conversion
pattern is simply not taken into account:
https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263
There should be room for an improvement here: would you mind to file an
issue on JIRA about propagating Date values as formatted strings?
> In that sense, it is better that it fails the other way around,
> because we are to fetch the values from the HR and they should not be
> managed from the syncope environment.
>
If such values are not to be used anywhere else in Syncope, then simply
define the internal schema as String rather than Date.
> *Second question:*
>
> When a connector has the SEARCH and SYNC operations associated with
> itself: how is the logic supposed to work if you do have the change
> log column set and deletes a tuple in the external database? Pull mode
> is set to incremental. On our end here, even though we delete the
> tuple in the mssql hr simulation database, the associated account is
> not deleted within syncope. Not even with pull mode set to \u201cFull
> reconciliation\u201d funny enough.
>
> What I can see, the pull action handler is not logging anything if the
> pull mode is not set to full reconciliation either. Syncope and the db
> do have the associated key values set correctly etc. The change log
> column is of a datetime format also. Does this has something to do
> with it?
>
You need to consider that everything related to SYNC mostly depends on
how the actual connector implements such operation.
The db-table connector is possibly one of the simplest out there; among
its limitation, it is also not capable to feed Syncope with information
about deleted users.
I would suggest to use the more flexible Scripted SQL for the job;
you'll need, however, to provide your own Groovy scripts.
You might want to look at
https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql
as starting point; FYI, look how one of such sample scripts can generate
DELETE events to send back to Syncope:
https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78
by simply considering an additional 'deleted' column.
--
Francesco Chicchiricc�
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/