You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ofbiz.apache.org by darrell73s <da...@gmail.com> on 2015/02/04 23:35:58 UTC

Offline EntitySync Functionality

Hi folks,

I'm exploring the use of the entity sync feature of the framework to perform
some data replication. Because of my setup, I would need to use the "offline
sync" portion of the functionality.

I have gone through the existing services, and the interface for offline
entity sync in webtools. However, when it comes time to "Accept Offline
Sync" or "Reject Offline Sync", a call is made to the
updateOfflineEntitySync service which is not yet implemented.

Based on the existing flow, I would think that at the very least, an
implemented updateOfflineEntitySync would update the runStatusId on the
proper EntitySync/EntitySyncHistory record. Is it envisioned that there
would be any additional logic (outside of what I've mentioned) in this
service to handle the Accept/Reject of an offline sync operation?

If I decide to go down this route, I wanted to ensure that I understood the
intended function of this service in case I wind up implementing something,
and/or contributing an implementation for this service.

Thanks!
Darrell



--
View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586.html
Sent from the OFBiz - User mailing list archive at Nabble.com.

Re: Offline EntitySync Functionality

Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi Darrel,

I have not used the EntitySync since fall 2009. Finally, I have maybe then used the "Offline EntitySync Functionality" but I can't clearly remember (I 
remember I had issue with the automated EntitySync and had to get my hands dirty by indeed using Webtools and I remember it then helped).

On a trunk HEAD local instance, I have just tried to push the "Run Offline Sync" button on the 1st line (Id=1500) and got this error

2015-02-09 07:54:11,057 |OFBiz-JobQueue-1 |EntitySyncContext             |I| Creating EntitySyncContext with entitySyncId=1500
2015-02-09 07:54:11,124 |OFBiz-JobQueue-1 |EntitySyncContext             |I| In makeEntityModelToUseList for EntitySync with ID [1500] syncing 837 
entities
2015-02-09 07:54:11,124 |OFBiz-JobQueue-1 |UtilMisc                      |I| null
java.lang.IllegalArgumentException: You must pass an even sized array to the toMap method (size = 3)
         at org.ofbiz.base.util.UtilMisc.toMap(UtilMisc.java:158) [ofbiz-base.jar:?]
         at org.ofbiz.entity.util.EntityQuery.where(EntityQuery.java:160) [ofbiz-entity.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncContext.getCurrentRunStartTime(EntitySyncContext.java:894) [ofbiz-entityext.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncContext.<init>(EntitySyncContext.java:188) [ofbiz-entityext.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncServices.runOfflineEntitySync(EntitySyncServices.java:451) [ofbiz-entityext.jar:?]

Same with the 2nd line (5500), etc.

So clearly we need to revisit either OOTB data or code (seems more data since you seem to got it working, were you using the trunk?)

This said about your questions, please read inline

Le 07/02/2015 00:18, darrell73s a écrit :
> Thanks Jacques,
>
> I had found that wiki article previously, which mainly documents the
> pull/push processes, and found the concepts helpful while going through the
> EntitySync source.
>
> However, as you've indicated, what I'm looking at is the offline sync.
> Please let me know if the following would be better suited for the "dev"
> mailing list.
>
> Assuming the scenario of an offline sync between the A (source) and B
> (target) nodes, here is how I believe it would occur between the two nodes
> (using the as-is WebTools flow in my example). Please feel free to correct
> my understanding, or if this scenario has already been hashed out, to point
> me in the right direction.
>
> 1) On Node A, user clicks "Run Offline Sync" which runs the
> 'runOfflineEntitySync' service. This service changes status on the
> EntitySync record, creates an EntitySyncHistory record, and produces an xml
> export of the data to be sync'd.
>
> 2) User on Node B inputs the xml file under "Load Offline Data" which
> ultimately runs 'loadOfflineSyncData' service. This service loads the
> exported data into the database by calling 'storeEntitySyncData'. There is a
> TODO in the code to "create a response document to send back to the initial
> sync machine".
>
> 3) Once there has been confirmation that the data was loaded into B, user
> presses "Accept Offline Sync" or "Reject Offline Sync" on Node A, which runs
> the 'updateOfflineEntitySync' service.
>
> Questions/Thoughts based on the above:
>
> * 'runOfflineEntitySync' service appears to be fully implemented
>
> * 'loadOfflineSyncData' service appears to be almost fully implemented:
>      - Service could be updated to produce the 'response document' mentioned
> in the TODO in cases where you may want to automate the third step in the
> process.

That sounds good to me

>      - This service makes a call to 'storeEntitySyncData' which produces a
> series of stats related to records created/updated/deleted. Nothing is done
> with these stats such as putting them into EntitySyncHistory because no
> EntitySyncHistory is created by 'loadOfflineSyncData'.

 From the top of head I have no ideas about that

> * 'updateOfflineEntitySync' service is NOT implemented

It should be, but you see not much people use it, it's there waiting for more than 10 years.

> 1) In 'loadOfflineSyncData', would it make sense to call
> EntitySyncContext.runOfflineStartRunning() which would mark the status of
> the EntitySync record as ESR_RUNNING, as well as create the initial
> EntitySyncHistory record?

I would check using ESR_RUNNING there has no side effects on current code before doing this.

> Then, we could populate the stats returned by
> 'storeEntitySyncData' such that there is record of the offline load
> occurring on node B (target node), and its results into the
> EntitySyncHistory record we've just created, perhaps by running
> EntitySyncContext.saveResultsReportedFromDataStore()?

Same as above, we must be cautious about how theEntitySync mechanism works

> At this point in the process, we could also export a 'response document'
> with the results of the sync.

+1

> 2) If 'updateOfflineEntitySync' were to be implemented, does it make sense
> for this service to: set the status of the EntitySync and corresponding
> EntitySyncHistory record based on whether: "Accept Offline Sync"/"Reject
> Offline Sync" button is pressed or optionally based on the response as sent
> in a 'response document'?

This would need more review on my side. Since you are interested, I suggest you
1) create a Jira to fix the current issue with OOTB data, with a patch if possible. We could then use that for testing things
2) create Jiras with patches for the other features and improvements you suggest

Thanks!

Jacques

>
> Thanks,
> Darrell
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586p4662652.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>

Re: Offline EntitySync Functionality

Posted by darrell73s <da...@gmail.com>.
Thanks Jacques, 

I had found that wiki article previously, which mainly documents the
pull/push processes, and found the concepts helpful while going through the
EntitySync source.

However, as you've indicated, what I'm looking at is the offline sync.
Please let me know if the following would be better suited for the "dev"
mailing list.

Assuming the scenario of an offline sync between the A (source) and B
(target) nodes, here is how I believe it would occur between the two nodes
(using the as-is WebTools flow in my example). Please feel free to correct
my understanding, or if this scenario has already been hashed out, to point
me in the right direction.

1) On Node A, user clicks "Run Offline Sync" which runs the
'runOfflineEntitySync' service. This service changes status on the
EntitySync record, creates an EntitySyncHistory record, and produces an xml
export of the data to be sync'd.

2) User on Node B inputs the xml file under "Load Offline Data" which
ultimately runs 'loadOfflineSyncData' service. This service loads the
exported data into the database by calling 'storeEntitySyncData'. There is a
TODO in the code to "create a response document to send back to the initial
sync machine".

3) Once there has been confirmation that the data was loaded into B, user
presses "Accept Offline Sync" or "Reject Offline Sync" on Node A, which runs
the 'updateOfflineEntitySync' service.

Questions/Thoughts based on the above:

* 'runOfflineEntitySync' service appears to be fully implemented

* 'loadOfflineSyncData' service appears to be almost fully implemented:
    - Service could be updated to produce the 'response document' mentioned
in the TODO in cases where you may want to automate the third step in the
process. 
    - This service makes a call to 'storeEntitySyncData' which produces a
series of stats related to records created/updated/deleted. Nothing is done
with these stats such as putting them into EntitySyncHistory because no
EntitySyncHistory is created by 'loadOfflineSyncData'. 

* 'updateOfflineEntitySync' service is NOT implemented 
	
1) In 'loadOfflineSyncData', would it make sense to call
EntitySyncContext.runOfflineStartRunning() which would mark the status of
the EntitySync record as ESR_RUNNING, as well as create the initial
EntitySyncHistory record? Then, we could populate the stats returned by
'storeEntitySyncData' such that there is record of the offline load
occurring on node B (target node), and its results into the
EntitySyncHistory record we've just created, perhaps by running
EntitySyncContext.saveResultsReportedFromDataStore()? 

At this point in the process, we could also export a 'response document'
with the results of the sync.

2) If 'updateOfflineEntitySync' were to be implemented, does it make sense
for this service to: set the status of the EntitySync and corresponding
EntitySyncHistory record based on whether: "Accept Offline Sync"/"Reject
Offline Sync" button is pressed or optionally based on the response as sent
in a 'response document'?

Thanks,
Darrell



--
View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586p4662652.html
Sent from the OFBiz - User mailing list archive at Nabble.com.

Re: Offline EntitySync Functionality

Posted by Jacques Le Roux <ja...@les7arts.com>.
This might help you figure out how entity sync works.

https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example

I have though never considered/needed what you call the "offline part"...

Jacques

Le 04/02/2015 23:35, darrell73s a écrit :
> Hi folks,
>
> I'm exploring the use of the entity sync feature of the framework to perform
> some data replication. Because of my setup, I would need to use the "offline
> sync" portion of the functionality.
>
> I have gone through the existing services, and the interface for offline
> entity sync in webtools. However, when it comes time to "Accept Offline
> Sync" or "Reject Offline Sync", a call is made to the
> updateOfflineEntitySync service which is not yet implemented.
>
> Based on the existing flow, I would think that at the very least, an
> implemented updateOfflineEntitySync would update the runStatusId on the
> proper EntitySync/EntitySyncHistory record. Is it envisioned that there
> would be any additional logic (outside of what I've mentioned) in this
> service to handle the Accept/Reject of an offline sync operation?
>
> If I decide to go down this route, I wanted to ensure that I understood the
> intended function of this service in case I wind up implementing something,
> and/or contributing an implementation for this service.
>
> Thanks!
> Darrell
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>