You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ofbiz.apache.org by Deyan Tsvetanov <de...@ittconsult.com> on 2009/10/04 11:57:29 UTC
EntitySync issues & suggestions ( was EntitySync RMI error )
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature I've come
up with the conclusion that it still needs
some work to get ready for production environment. Currently it is
unreliable, and useless :(
- There is a bug in the design. The MCS remembers the sync status and
currently reached sync timestamp for each client.
This is not a good approach imho. Why ?
The client asks the server: have you got something for me ? Next: The
MCS checks the last successful sync timestamp for that particular client
and selects the entities that should be passed to the client. After
successfully transferring the entities to the client over RMI the MCS
assumes that the sync is successful and updates the last successful sync
timestamp.
The problem is that the client does not confirm the success of the
operation. If for some reason the client is unable to update the
entities ( either shutdown,
power failure, DB error or whatever ) the MCS isn't aware of that and
the next sync will not include the missed entities. There is no way to
fix the skipped
entities but re-creating the client's database from zero.
That is the main issue actually. It could be easily solved however. The
most easy ways are:
1) The client will remember the sync timestamp and will include it as an
argument when asking the server for pull sync. This way the client will
update the last successful sync timestamp after storing each entity. The
server does not have to remember anything regarding the client's sync
status.
2) After each sync session the server will wait for confirmation from
the client before updating the last successful sync timestamp.
If not updated than the sync process is considered to be unsuccessful
and the timestamp will not be updated.
I personally prefer approach 1) as in case of network error, power
failure or whatever error the sync process will continue from the place
it has stopped.
- There are some swallowed exceptions. If a network problem occurs
during the sync process the server's status remains "sync running" and
the client is
not able to sync anymore until the status is reset by an operator.
- The last issue is the performance. Using RMI is an easy approach from
developer's perspective but it is very very slow.
I had a database with 5000 products. The initial sync took about 2
hours. The hardware is pretty good, the network connection between the
MCS and the client ofbiz
is about 50 mbits. What if we have 100 000 sales per day ? The sync
process will take probably the entire night :)
Here I would suggest transferring GenericEntity instances over RMI to be
removed. Instead a regular XML could be used. Either over RMI or SOAP -
does not matter.
So in general the picture would be:
1) Server does not remember any timestamp for any client.
2) Client calls the pull entity sync RMI ( or soap ) method and passes
the entity sync ID and the timestamp of the last successful sync as
arguments. Including the timezone of course :)
3) The server generates an XML file for all the entities in that entity
sync group using the timestamp provided by the client.
4) Client starts updating the entities described in the received XML
line by line. After each entity is updated successfully the timestamp is
also updated ( on the client side - the
server does not remember any client timestamps ).
5) If a failure occurs in the middle of the update process what we'll
have is a sync process completed to like 50%. With the correct
timestamp. When the next sync
job starts the client will use the timestamp and the server will
generate a new XML from the place the sync has failed. Of course an
optimization could be used -
the XML will be stored in a new db table so the client can continue
storing the entities without asking the server to generate something he
has already done.
Some improvements that could be also useful:
1) The client ofbiz instances have sequence-id-prefix. So when the
server is accepting push syncs it could also check the prefixes of the
PKs and disallow entities without the correct
prefix in the primary key. That's just a cheap insurance against
errors :) We don't want store A to make sales on behalf of store B just
because somebody made an error while configuring the
pos ofbiz instance.
2) A time synchronization mechanism could be also implemented. In
general when syncing there is a requirement that the server's and
client's clocks are in sync ( ntp ). We don't want
to set the system clock in ofbiz, but we could implement a check if the
clocks are in sync and refuse to start the synchronization in certain
cases.
3) A new column could be added to all the entities: isFromEntitySync :
flag. This way we could avoid cases in which the client is trying to
push records to the server which were
actually pulled by the server. Yes, I know the entity's created
timestamp should help us avoid such cases, but when the sync process is
interrupted things like that could happen.
Otherwise for our current project we've implemented a DB level
synchronization - using triggers for INSERT, UPDATE and DELETE and a
stored procedure which syncs the clients with a MCS.
It works much ( tens of times ) faster than ofbiz entity sync . But it
is DB dependent. So after getting comments and suggestions to this
e-mail I would like to invest some time in
fixing all those issues with the ofbiz entity sync and switch to it when
it's ready :)
Thanks for your time,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: user@ofbiz.apache.org
Subject: Re: EntitySync RMI error
Date: Sun, 27 Sep 2009 17:02:57 +0200
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I
will eventually take care to put it under FAQ...
TIA
Jacques
From: "Deyan Tsvetanov" <de...@ittconsult.com>
>I found an issue though:
>
> If the connection gets dropped during sync :
> - the client ( POS ) prints connection reset by peer
> - the server ( MCS ) sync status remains to running.
> - the next sync does not start because MCS complains there is another
> sync running already.
>
> "Reset run status" from webtools -> entity sync status
> helps.
>
>
> I'll investigate further and log a bug, although the solution seems
> pretty simple - looks like a swallowed IOException,
> which should be handled by re-setting the sync status.
>
> CHeers,
> DEyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 11:22:55 +0300
>
>
> Got it !
>
> Seems that the previous exception was coming from the MCS ( server ) .
> When I stopped the server I started getting the exception I kind of like
> more :)
>
>
> Message: Exception calling remote pull and report EntitySync service
> with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> ricServiceException: RMI Error (Connection refused to host:
> 192.168.1.100; nested exception is:
> java.net.ConnectException: Connection refused: connect)
>
> So the problem was at the server side - RMIIF env var in startofbiz.sh
> wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> hostname .
>
> The misleading things for me was that:
> 1) no any sign in the MCS logs that somebody is trying to connect and
> the MCS itself can not connect to its own RMI registry.
> 2) no any difference in the exceptions text on the POS ( client ) side:
> the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> remote one - can not connect to 127.0.0.1 )
> look the same :)
>
> So issue is solved,
> sorry for bothering :)
>
> -- deyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:52:55 +0300
>
>
> During startup I get:
>
> 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> [main-rmi=rmi://127.0.0.
> 1:1099/RMIDispatcher,
> main-http=http://127.0.0.1:8080/webtools/control/httpService,
> entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> 0.0.1:8080/webtools/control/httpService]
>
> entity-sync-rmi seems to be ok ...
>
>
>
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:22:23 +0300
>
>
> Hi guys,
>
> I'm trying to configure RMI entity sync. I'm following
> http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> What I've done so far:
>
> 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>
> 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> ( as per the example ).
>
> 3) I've imported the entity sync groups, SandJobs, etc.
>
> However when the sync starts ( on the POS instance ) I get the following
> error:
>
> Exception calling remote pull and report EntitySync service with name:
> remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> Exception: org.ofbiz.service.GenericServiceException
> Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.rmi.ConnectException
> Message: Connection refused to host: 127.0.0.1; nested exception is:
> java.net.ConnectException: Connection refused: connect
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.net.ConnectException
> Message: Connection refused: connect
> ---- stack trace
> ---------------------------------------------------------------
>
>
> It insists connecting to 127.0.0.1 no matter what I type in
> serviceengine.xml.
> Any help would be appreciated :)
>
> Thanks in advance,
> Deyan
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Adrian Crum <ad...@hlmksw.com>.
I don't use entity sync, and I haven't looked at the code, but based on
what I've read in the mailing lists, I have some ideas. Take them for
what they are worth.
If the problems mentioned in this thread and others really do exist, and
if there is a consensus that the entity sync code needs to be reworked,
then why not find an alternative to RMI - since there seems to be a
performance problem there?
It seems to me we could develop an XML file format that describes the
update tasks needed, and another XML file format that describes the
final outcome of each update task. The XML files could be zipped and
transferred using regular file transfer protocols.
It would be similar to how WebDAV works. Maybe we could even create a
WebDAV interface for entity synch.
-Adrian
Jacques Le Roux wrote:
> Hi Deyan, Pankaj,
>
> Could you share your work done so far with the community ?
> I agree that we shouyd try to use and maintain EntitySync.
> But for the volume reason Deyan outlined and in the meantime I'm sure
> his solution would be appreciated too!
> I suggest to create Jira or/and wiki pages. I'm ready to help on this.
>
> Thanks
>
> Jacques
>
> From: "Jacques Le Roux" <ja...@les7arts.com>
>> Thanks for comment Deyan,
>>
>> I will try Pankaj Jain's tips in a 1st time
>>
>> Jacques
>>
>> From: "Deyan Tsvetanov" <de...@ittconsult.com>
>> Hi Jacques,
>>
>> no, I didn't work on fixing the entity sync issues.
>> Instead I did my own sync tool using db triggers and stored procedures.
>>
>> The entity sync could be fixed, but it would require like two - three
>> weeks of work.
>> I am sorry I didn't post any details, i was kind of busy because of a
>> missed deadline :)
>>
>> The entity sync is kind of useless in this state imho.
>> Reasons:
>>
>> 1) Does not do any fail over - if the sync process is interrupted for
>> any reason - like network outage, or ofbiz restart, or system freeze
>> it will be considered as a success sync and won't continue from the
>> point it has failed. Could be fixed though :)
>>
>> 2) Pretty slow. The entities are transfered over the network using RMI.
>> For transferring 5000 product records ( along with their relations -
>> product_price, product_assoc, etc )
>> it took over 2 hours using pretty good hard and 50 mbits network
>> connection.
>>
>> 3) The example of entities to sync in the sync howto notes will never
>> work. There will be records with duplicate PKs :)
>>
>> The solution for those issues imho are:
>> 1) Refactor a bit the entity sync code so the client remembers the last
>> successful sync state. This would fix the fail over issues. currently
>> the server ( MCS ) remembers the
>> sync status for each client.
>>
>> 2) Don't use RMI :) It would be much faster if the entities that are
>> going to be transfered in order to get synced are serialized to XML for
>> example and
>> transferred over the network in a text format. RMI is easy to use but
>> has bad performance. Just to give you an example:
>> about 5000 products ( which is probably about 20 000 entities in total
>> to pull ) takes 2 hours on a 50 mbits connection. What if we had a
>> pretty huge supermarket
>> which has 100 000 sales per day ( this means 100 000 sales orders ,
>> average 50 order_items per order, 100 000 payments, 100 000 invoices,
>> 100 000 invoice items ).
>> It would require an entire night to sync - if we are lucky :)
>>
>> 3) I can provide the complete list of tables needed to get synced for a
>> POS solution, including fix of the issue with the duplicate PKs :)
>>
>> However for our current project we've used triggers which create a
>> transaction log for push on the POS databases and a transaction log for
>> pull on the MCS instance.
>> Pulling those 5000 products along with all the related records takes
>> about 20 secs on the same hardware. I've configured a crontab job which
>> invoces the sync postgre stored procedures
>> every 5 minutes and it works perfectly. A status table is maintained on
>> the MCS so we can check regularly if a POS hasn't synced for let's say 1
>> hour - this means either network outage or some other problem.
>>
>> Yes, I know this solution is DB dependent but if the ofbiz sync
>> framework issues are fixed and we use the list of tables to sync i've
>> managed to create something useful could come out of it :)
>>
>> So - i hope my e-mail answers your question, if you want to know more -
>> don't hesitate :)
>>
>> Cheers,
>> Deyan
>>
>> -----Original Message-----
>> From: Jacques Le Roux <ja...@les7arts.com>
>> Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
>> To: Deyan Tsvetanov <de...@ittconsult.com>,
>> user@ofbiz.apache.org
>> Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
>> error )
>> Date: Mon, 26 Oct 2009 08:34:51 +0100
>>
>>
>> Hi Deyan,
>>
>> Did you work on this ? I will need to use EntitySync soon and I know you
>> are not the 1st to complain about it (Si Chen did for instance). On the
>> other hand, some seems to have used it successfully...
>>
>> Thanks
>>
>> Jacques
>> ----- Original Message ----- From: Deyan Tsvetanov
>> To: Jacques Le Roux ; user@ofbiz.apache.org
>> Sent: Sunday, October 04, 2009 10:57 AM
>> Subject: EntitySync issues & suggestions ( was EntitySync RMI
>> error )
>>
>>
>>
>> Hi Jacques and list,
>>
>> After playing for almost 2 weeks with the entity sync feature
>> I've come up with the conclusion that it still needs
>> some work to get ready for production environment. Currently it
>> is unreliable, and useless :(
>>
>> - There is a bug in the design. The MCS remembers the sync
>> status and currently reached sync timestamp for each client.
>> This is not a good approach imho. Why ?
>> The client asks the server: have you got something for me ?
>> Next: The MCS checks the last successful sync timestamp for that
>> particular client and selects the entities that should be passed
>> to the client. After successfully transferring the entities to
>> the client over RMI the MCS assumes that the sync is successful
>> and updates the last successful sync timestamp.
>>
>> The problem is that the client does not confirm the success of
>> the operation. If for some reason the client is unable to update
>> the entities ( either shutdown,
>> power failure, DB error or whatever ) the MCS isn't aware of
>> that and the next sync will not include the missed entities.
>> There is no way to fix the skipped
>> entities but re-creating the client's database from zero.
>>
>> That is the main issue actually. It could be easily solved
>> however. The most easy ways are:
>> 1) The client will remember the sync timestamp and will include
>> it as an argument when asking the server for pull sync. This way
>> the client will
>> update the last successful sync timestamp after storing each
>> entity. The server does not have to remember anything regarding
>> the client's sync status.
>>
>> 2) After each sync session the server will wait for confirmation
>> from the client before updating the last successful sync
>> timestamp.
>> If not updated than the sync process is considered to be
>> unsuccessful and the timestamp will not be updated.
>>
>> I personally prefer approach 1) as in case of network error,
>> power failure or whatever error the sync process will continue
>> from the place it has stopped.
>>
>>
>> - There are some swallowed exceptions. If a network problem
>> occurs during the sync process the server's status remains "sync
>> running" and the client is
>> not able to sync anymore until the status is reset by an
>> operator.
>>
>> - The last issue is the performance. Using RMI is an easy
>> approach from developer's perspective but it is very very slow.
>> I had a database with 5000 products. The initial sync took about
>> 2 hours. The hardware is pretty good, the network connection
>> between the MCS and the client ofbiz
>> is about 50 mbits. What if we have 100 000 sales per day ? The
>> sync process will take probably the entire night :)
>> Here I would suggest transferring GenericEntity instances over
>> RMI to be removed. Instead a regular XML could be used. Either
>> over RMI or SOAP - does not matter.
>>
>> So in general the picture would be:
>>
>> 1) Server does not remember any timestamp for any client.
>> 2) Client calls the pull entity sync RMI ( or soap ) method and
>> passes the entity sync ID and the timestamp of the last
>> successful sync as arguments. Including the timezone of
>> course :)
>> 3) The server generates an XML file for all the entities in that
>> entity sync group using the timestamp provided by the client.
>> 4) Client starts updating the entities described in the received
>> XML line by line. After each entity is updated successfully the
>> timestamp is also updated ( on the client side - the
>> server does not remember any client timestamps ).
>> 5) If a failure occurs in the middle of the update process what
>> we'll have is a sync process completed to like 50%. With the
>> correct timestamp. When the next sync
>> job starts the client will use the timestamp and the server will
>> generate a new XML from the place the sync has failed. Of course
>> an optimization could be used -
>> the XML will be stored in a new db table so the client can
>> continue storing the entities without asking the server to
>> generate something he has already done.
>>
>> Some improvements that could be also useful:
>> 1) The client ofbiz instances have sequence-id-prefix. So when
>> the server is accepting push syncs it could also check the
>> prefixes of the PKs and disallow entities without the correct
>> prefix in the primary key. That's just a cheap insurance against
>> errors :) We don't want store A to make sales on behalf of store
>> B just because somebody made an error while configuring the
>> pos ofbiz instance.
>>
>> 2) A time synchronization mechanism could be also implemented.
>> In general when syncing there is a requirement that the server's
>> and client's clocks are in sync ( ntp ). We don't want
>> to set the system clock in ofbiz, but we could implement a check
>> if the clocks are in sync and refuse to start the
>> synchronization in certain cases.
>>
>> 3) A new column could be added to all the entities:
>> isFromEntitySync : flag. This way we could avoid cases in which
>> the client is trying to push records to the server which were
>> actually pulled by the server. Yes, I know the entity's created
>> timestamp should help us avoid such cases, but when the sync
>> process is interrupted things like that could happen.
>>
>> Otherwise for our current project we've implemented a DB level
>> synchronization - using triggers for INSERT, UPDATE and DELETE
>> and a stored procedure which syncs the clients with a MCS.
>> It works much ( tens of times ) faster than ofbiz entity sync .
>> But it is DB dependent. So after getting comments and
>> suggestions to this e-mail I would like to invest some time in
>> fixing all those issues with the ofbiz entity sync and switch to
>> it when it's ready :)
>>
>> Thanks for your time,
>> Deyan
>>
>> -----Original Message-----
>> From: Jacques Le Roux <ja...@les7arts.com>
>> Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
>> To: user@ofbiz.apache.org
>> Subject: Re: EntitySync RMI error
>> Date: Sun, 27 Sep 2009 17:02:57 +0200
>>
>>
>> Hi Deyan,
>>
>> It would be very valuable if you could post an article on wiki
>> about your experience with this. Don't worry about where to put it, I
>> will eventually take care to put it under FAQ...
>>
>> TIA
>>
>> Jacques
>>
>> From: "Deyan Tsvetanov" <de...@ittconsult.com>
>> >I found an issue though:
>> >
>> > If the connection gets dropped during sync :
>> > - the client ( POS ) prints connection reset by peer
>> > - the server ( MCS ) sync status remains to running.
>> > - the next sync does not start because MCS complains there is
>> another
>> > sync running already.
>> >
>> > "Reset run status" from webtools -> entity sync status
>> > helps.
>> >
>> >
>> > I'll investigate further and log a bug, although the solution
>> seems
>> > pretty simple - looks like a swallowed IOException,
>> > which should be handled by re-setting the sync status.
>> >
>> > CHeers,
>> > DEyan
>> >
>> > -----Original Message-----
>> > From: Deyan Tsvetanov <de...@ittconsult.com>
>> > Reply-to: user@ofbiz.apache.org
>> > To: user@ofbiz.apache.org
>> > Subject: Re: EntitySync RMI error
>> > Date: Sat, 26 Sep 2009 11:22:55 +0300
>> >
>> >
>> > Got it !
>> >
>> > Seems that the previous exception was coming from the MCS (
>> server ) .
>> > When I stopped the server I started getting the exception I
>> kind of like
>> > more :)
>> >
>> >
>> > Message: Exception calling remote pull and report EntitySync
>> service
>> > with name: remotePullAndReportEntitySyncDataRmi;
>> org.ofbiz.service.Gene
>> > ricServiceException: RMI Error (Connection refused to host:
>> > 192.168.1.100; nested exception is:
>> > java.net.ConnectException: Connection refused: connect)
>> >
>> > So the problem was at the server side - RMIIF env var in
>> startofbiz.sh
>> > wasn't set , so ofbiz was trying to get RMI host IP by
>> resolving the
>> > hostname .
>> >
>> > The misleading things for me was that:
>> > 1) no any sign in the MCS logs that somebody is trying to
>> connect and
>> > the MCS itself can not connect to its own RMI registry.
>> > 2) no any difference in the exceptions text on the POS (
>> client ) side:
>> > the 2 exceptions ( local one - can not connect to
>> 192.168.1.100 and the
>> > remote one - can not connect to 127.0.0.1 )
>> > look the same :)
>> >
>> > So issue is solved,
>> > sorry for bothering :)
>> >
>> > -- deyan
>> >
>> > -----Original Message-----
>> > From: Deyan Tsvetanov <de...@ittconsult.com>
>> > Reply-to: user@ofbiz.apache.org
>> > To: user@ofbiz.apache.org
>> > Subject: Re: EntitySync RMI error
>> > Date: Sat, 26 Sep 2009 09:52:55 +0300
>> >
>> >
>> > During startup I get:
>> >
>> > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
>> > AbstractEngine.java:73 :INFO ] Loaded Service Locations :
>> > [main-rmi=rmi://127.0.0.
>> > 1:1099/RMIDispatcher,
>> > main-http=http://127.0.0.1:8080/webtools/control/httpService,
>> > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
>> >
>> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
>> > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher,
>> eedcc-test=http://127.
>> > 0.0.1:8080/webtools/control/httpService]
>> >
>> > entity-sync-rmi seems to be ok ...
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Deyan Tsvetanov <de...@ittconsult.com>
>> > Reply-to: user@ofbiz.apache.org
>> > To: user@ofbiz.apache.org
>> > Subject: EntitySync RMI error
>> > Date: Sat, 26 Sep 2009 09:22:23 +0300
>> >
>> >
>> > Hi guys,
>> >
>> > I'm trying to configure RMI entity sync. I'm following
>> > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>> >
>> > What I've done so far:
>> >
>> > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>> >
>> > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
>> > ( as per the example ).
>> >
>> > 3) I've imported the entity sync groups, SandJobs, etc.
>> >
>> > However when the sync starts ( on the POS instance ) I get
>> the following
>> > error:
>> >
>> > Exception calling remote pull and report EntitySync service
>> with name:
>> > remotePullAndReportEntitySyncDataRmi;
>> org.ofbiz.service.GenericServic
>> > eException: RMI Invocation Error (Connection refused to host:
>> 127.0.0.1;
>> > nested exception is:
>> > java.net.ConnectException: Connection refused: connect)
>> > Exception: org.ofbiz.service.GenericServiceException
>> > Message: RMI Invocation Error (Connection refused to host:
>> 127.0.0.1;
>> > nested exception is:
>> > java.net.ConnectException: Connection refused: connect)
>> > ---- cause
>> >
>> ---------------------------------------------------------------------
>> > Exception: java.rmi.ConnectException
>> > Message: Connection refused to host: 127.0.0.1; nested
>> exception is:
>> > java.net.ConnectException: Connection refused: connect
>> > ---- cause
>> >
>> ---------------------------------------------------------------------
>> > Exception: java.net.ConnectException
>> > Message: Connection refused: connect
>> > ---- stack trace
>> > ---------------------------------------------------------------
>> >
>> >
>> > It insists connecting to 127.0.0.1 no matter what I type in
>> > serviceengine.xml.
>> > Any help would be appreciated :)
>> >
>> > Thanks in advance,
>> > Deyan
>> >
>>
>>
>>
>>
>
>
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Hans Bakker <ma...@antwebsystems.com>.
I have a customer who is also interested...
Hans
On Tue, 2009-10-27 at 08:58 +0100, Jacques Le Roux wrote:
> Hi Deyan, Pankaj,
>
> Could you share your work done so far with the community ?
> I agree that we shouyd try to use and maintain EntitySync.
> But for the volume reason Deyan outlined and in the meantime I'm sure his solution would be appreciated too!
> I suggest to create Jira or/and wiki pages. I'm ready to help on this.
>
> Thanks
>
> Jacques
>
> From: "Jacques Le Roux" <ja...@les7arts.com>
> > Thanks for comment Deyan,
> >
> > I will try Pankaj Jain's tips in a 1st time
> >
> > Jacques
> >
> > From: "Deyan Tsvetanov" <de...@ittconsult.com>
> > Hi Jacques,
> >
> > no, I didn't work on fixing the entity sync issues.
> > Instead I did my own sync tool using db triggers and stored procedures.
> >
> > The entity sync could be fixed, but it would require like two - three
> > weeks of work.
> > I am sorry I didn't post any details, i was kind of busy because of a
> > missed deadline :)
> >
> > The entity sync is kind of useless in this state imho.
> > Reasons:
> >
> > 1) Does not do any fail over - if the sync process is interrupted for
> > any reason - like network outage, or ofbiz restart, or system freeze
> > it will be considered as a success sync and won't continue from the
> > point it has failed. Could be fixed though :)
> >
> > 2) Pretty slow. The entities are transfered over the network using RMI.
> > For transferring 5000 product records ( along with their relations -
> > product_price, product_assoc, etc )
> > it took over 2 hours using pretty good hard and 50 mbits network
> > connection.
> >
> > 3) The example of entities to sync in the sync howto notes will never
> > work. There will be records with duplicate PKs :)
> >
> > The solution for those issues imho are:
> > 1) Refactor a bit the entity sync code so the client remembers the last
> > successful sync state. This would fix the fail over issues. currently
> > the server ( MCS ) remembers the
> > sync status for each client.
> >
> > 2) Don't use RMI :) It would be much faster if the entities that are
> > going to be transfered in order to get synced are serialized to XML for
> > example and
> > transferred over the network in a text format. RMI is easy to use but
> > has bad performance. Just to give you an example:
> > about 5000 products ( which is probably about 20 000 entities in total
> > to pull ) takes 2 hours on a 50 mbits connection. What if we had a
> > pretty huge supermarket
> > which has 100 000 sales per day ( this means 100 000 sales orders ,
> > average 50 order_items per order, 100 000 payments, 100 000 invoices,
> > 100 000 invoice items ).
> > It would require an entire night to sync - if we are lucky :)
> >
> > 3) I can provide the complete list of tables needed to get synced for a
> > POS solution, including fix of the issue with the duplicate PKs :)
> >
> > However for our current project we've used triggers which create a
> > transaction log for push on the POS databases and a transaction log for
> > pull on the MCS instance.
> > Pulling those 5000 products along with all the related records takes
> > about 20 secs on the same hardware. I've configured a crontab job which
> > invoces the sync postgre stored procedures
> > every 5 minutes and it works perfectly. A status table is maintained on
> > the MCS so we can check regularly if a POS hasn't synced for let's say 1
> > hour - this means either network outage or some other problem.
> >
> > Yes, I know this solution is DB dependent but if the ofbiz sync
> > framework issues are fixed and we use the list of tables to sync i've
> > managed to create something useful could come out of it :)
> >
> > So - i hope my e-mail answers your question, if you want to know more -
> > don't hesitate :)
> >
> > Cheers,
> > Deyan
> >
> > -----Original Message-----
> > From: Jacques Le Roux <ja...@les7arts.com>
> > Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
> > To: Deyan Tsvetanov <de...@ittconsult.com>,
> > user@ofbiz.apache.org
> > Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
> > error )
> > Date: Mon, 26 Oct 2009 08:34:51 +0100
> >
> >
> > Hi Deyan,
> >
> > Did you work on this ? I will need to use EntitySync soon and I know you
> > are not the 1st to complain about it (Si Chen did for instance). On the
> > other hand, some seems to have used it successfully...
> >
> > Thanks
> >
> > Jacques
> > ----- Original Message -----
> > From: Deyan Tsvetanov
> > To: Jacques Le Roux ; user@ofbiz.apache.org
> > Sent: Sunday, October 04, 2009 10:57 AM
> > Subject: EntitySync issues & suggestions ( was EntitySync RMI
> > error )
> >
> >
> >
> > Hi Jacques and list,
> >
> > After playing for almost 2 weeks with the entity sync feature
> > I've come up with the conclusion that it still needs
> > some work to get ready for production environment. Currently it
> > is unreliable, and useless :(
> >
> > - There is a bug in the design. The MCS remembers the sync
> > status and currently reached sync timestamp for each client.
> > This is not a good approach imho. Why ?
> > The client asks the server: have you got something for me ?
> > Next: The MCS checks the last successful sync timestamp for that
> > particular client and selects the entities that should be passed
> > to the client. After successfully transferring the entities to
> > the client over RMI the MCS assumes that the sync is successful
> > and updates the last successful sync timestamp.
> >
> > The problem is that the client does not confirm the success of
> > the operation. If for some reason the client is unable to update
> > the entities ( either shutdown,
> > power failure, DB error or whatever ) the MCS isn't aware of
> > that and the next sync will not include the missed entities.
> > There is no way to fix the skipped
> > entities but re-creating the client's database from zero.
> >
> > That is the main issue actually. It could be easily solved
> > however. The most easy ways are:
> > 1) The client will remember the sync timestamp and will include
> > it as an argument when asking the server for pull sync. This way
> > the client will
> > update the last successful sync timestamp after storing each
> > entity. The server does not have to remember anything regarding
> > the client's sync status.
> >
> > 2) After each sync session the server will wait for confirmation
> > from the client before updating the last successful sync
> > timestamp.
> > If not updated than the sync process is considered to be
> > unsuccessful and the timestamp will not be updated.
> >
> > I personally prefer approach 1) as in case of network error,
> > power failure or whatever error the sync process will continue
> > from the place it has stopped.
> >
> >
> > - There are some swallowed exceptions. If a network problem
> > occurs during the sync process the server's status remains "sync
> > running" and the client is
> > not able to sync anymore until the status is reset by an
> > operator.
> >
> > - The last issue is the performance. Using RMI is an easy
> > approach from developer's perspective but it is very very slow.
> > I had a database with 5000 products. The initial sync took about
> > 2 hours. The hardware is pretty good, the network connection
> > between the MCS and the client ofbiz
> > is about 50 mbits. What if we have 100 000 sales per day ? The
> > sync process will take probably the entire night :)
> > Here I would suggest transferring GenericEntity instances over
> > RMI to be removed. Instead a regular XML could be used. Either
> > over RMI or SOAP - does not matter.
> >
> > So in general the picture would be:
> >
> > 1) Server does not remember any timestamp for any client.
> > 2) Client calls the pull entity sync RMI ( or soap ) method and
> > passes the entity sync ID and the timestamp of the last
> > successful sync as arguments. Including the timezone of
> > course :)
> > 3) The server generates an XML file for all the entities in that
> > entity sync group using the timestamp provided by the client.
> > 4) Client starts updating the entities described in the received
> > XML line by line. After each entity is updated successfully the
> > timestamp is also updated ( on the client side - the
> > server does not remember any client timestamps ).
> > 5) If a failure occurs in the middle of the update process what
> > we'll have is a sync process completed to like 50%. With the
> > correct timestamp. When the next sync
> > job starts the client will use the timestamp and the server will
> > generate a new XML from the place the sync has failed. Of course
> > an optimization could be used -
> > the XML will be stored in a new db table so the client can
> > continue storing the entities without asking the server to
> > generate something he has already done.
> >
> > Some improvements that could be also useful:
> > 1) The client ofbiz instances have sequence-id-prefix. So when
> > the server is accepting push syncs it could also check the
> > prefixes of the PKs and disallow entities without the correct
> > prefix in the primary key. That's just a cheap insurance against
> > errors :) We don't want store A to make sales on behalf of store
> > B just because somebody made an error while configuring the
> > pos ofbiz instance.
> >
> > 2) A time synchronization mechanism could be also implemented.
> > In general when syncing there is a requirement that the server's
> > and client's clocks are in sync ( ntp ). We don't want
> > to set the system clock in ofbiz, but we could implement a check
> > if the clocks are in sync and refuse to start the
> > synchronization in certain cases.
> >
> > 3) A new column could be added to all the entities:
> > isFromEntitySync : flag. This way we could avoid cases in which
> > the client is trying to push records to the server which were
> > actually pulled by the server. Yes, I know the entity's created
> > timestamp should help us avoid such cases, but when the sync
> > process is interrupted things like that could happen.
> >
> > Otherwise for our current project we've implemented a DB level
> > synchronization - using triggers for INSERT, UPDATE and DELETE
> > and a stored procedure which syncs the clients with a MCS.
> > It works much ( tens of times ) faster than ofbiz entity sync .
> > But it is DB dependent. So after getting comments and
> > suggestions to this e-mail I would like to invest some time in
> > fixing all those issues with the ofbiz entity sync and switch to
> > it when it's ready :)
> >
> > Thanks for your time,
> > Deyan
> >
> > -----Original Message-----
> > From: Jacques Le Roux <ja...@les7arts.com>
> > Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
> > To: user@ofbiz.apache.org
> > Subject: Re: EntitySync RMI error
> > Date: Sun, 27 Sep 2009 17:02:57 +0200
> >
> >
> > Hi Deyan,
> >
> > It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to
> > put it, I
> > will eventually take care to put it under FAQ...
> >
> > TIA
> >
> > Jacques
> >
> > From: "Deyan Tsvetanov" <de...@ittconsult.com>
> > >I found an issue though:
> > >
> > > If the connection gets dropped during sync :
> > > - the client ( POS ) prints connection reset by peer
> > > - the server ( MCS ) sync status remains to running.
> > > - the next sync does not start because MCS complains there is another
> > > sync running already.
> > >
> > > "Reset run status" from webtools -> entity sync status
> > > helps.
> > >
> > >
> > > I'll investigate further and log a bug, although the solution seems
> > > pretty simple - looks like a swallowed IOException,
> > > which should be handled by re-setting the sync status.
> > >
> > > CHeers,
> > > DEyan
> > >
> > > -----Original Message-----
> > > From: Deyan Tsvetanov <de...@ittconsult.com>
> > > Reply-to: user@ofbiz.apache.org
> > > To: user@ofbiz.apache.org
> > > Subject: Re: EntitySync RMI error
> > > Date: Sat, 26 Sep 2009 11:22:55 +0300
> > >
> > >
> > > Got it !
> > >
> > > Seems that the previous exception was coming from the MCS ( server ) .
> > > When I stopped the server I started getting the exception I kind of like
> > > more :)
> > >
> > >
> > > Message: Exception calling remote pull and report EntitySync service
> > > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> > > ricServiceException: RMI Error (Connection refused to host:
> > > 192.168.1.100; nested exception is:
> > > java.net.ConnectException: Connection refused: connect)
> > >
> > > So the problem was at the server side - RMIIF env var in startofbiz.sh
> > > wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> > > hostname .
> > >
> > > The misleading things for me was that:
> > > 1) no any sign in the MCS logs that somebody is trying to connect and
> > > the MCS itself can not connect to its own RMI registry.
> > > 2) no any difference in the exceptions text on the POS ( client ) side:
> > > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> > > remote one - can not connect to 127.0.0.1 )
> > > look the same :)
> > >
> > > So issue is solved,
> > > sorry for bothering :)
> > >
> > > -- deyan
> > >
> > > -----Original Message-----
> > > From: Deyan Tsvetanov <de...@ittconsult.com>
> > > Reply-to: user@ofbiz.apache.org
> > > To: user@ofbiz.apache.org
> > > Subject: Re: EntitySync RMI error
> > > Date: Sat, 26 Sep 2009 09:52:55 +0300
> > >
> > >
> > > During startup I get:
> > >
> > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> > > AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> > > [main-rmi=rmi://127.0.0.
> > > 1:1099/RMIDispatcher,
> > > main-http=http://127.0.0.1:8080/webtools/control/httpService,
> > > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> > > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> > > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> > > 0.0.1:8080/webtools/control/httpService]
> > >
> > > entity-sync-rmi seems to be ok ...
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Deyan Tsvetanov <de...@ittconsult.com>
> > > Reply-to: user@ofbiz.apache.org
> > > To: user@ofbiz.apache.org
> > > Subject: EntitySync RMI error
> > > Date: Sat, 26 Sep 2009 09:22:23 +0300
> > >
> > >
> > > Hi guys,
> > >
> > > I'm trying to configure RMI entity sync. I'm following
> > > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
> > >
> > > What I've done so far:
> > >
> > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
> > >
> > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> > > ( as per the example ).
> > >
> > > 3) I've imported the entity sync groups, SandJobs, etc.
> > >
> > > However when the sync starts ( on the POS instance ) I get the following
> > > error:
> > >
> > > Exception calling remote pull and report EntitySync service with name:
> > > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> > > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> > > nested exception is:
> > > java.net.ConnectException: Connection refused: connect)
> > > Exception: org.ofbiz.service.GenericServiceException
> > > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> > > nested exception is:
> > > java.net.ConnectException: Connection refused: connect)
> > > ---- cause
> > > ---------------------------------------------------------------------
> > > Exception: java.rmi.ConnectException
> > > Message: Connection refused to host: 127.0.0.1; nested exception is:
> > > java.net.ConnectException: Connection refused: connect
> > > ---- cause
> > > ---------------------------------------------------------------------
> > > Exception: java.net.ConnectException
> > > Message: Connection refused: connect
> > > ---- stack trace
> > > ---------------------------------------------------------------
> > >
> > >
> > > It insists connecting to 127.0.0.1 no matter what I type in
> > > serviceengine.xml.
> > > Any help would be appreciated :)
> > >
> > > Thanks in advance,
> > > Deyan
> > >
> >
> >
> >
> >
>
>
--
Antwebsystems.com: Quality OFBiz services for competitive rates
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi Deyan, Pankaj,
Could you share your work done so far with the community ?
I agree that we shouyd try to use and maintain EntitySync.
But for the volume reason Deyan outlined and in the meantime I'm sure his solution would be appreciated too!
I suggest to create Jira or/and wiki pages. I'm ready to help on this.
Thanks
Jacques
From: "Jacques Le Roux" <ja...@les7arts.com>
> Thanks for comment Deyan,
>
> I will try Pankaj Jain's tips in a 1st time
>
> Jacques
>
> From: "Deyan Tsvetanov" <de...@ittconsult.com>
> Hi Jacques,
>
> no, I didn't work on fixing the entity sync issues.
> Instead I did my own sync tool using db triggers and stored procedures.
>
> The entity sync could be fixed, but it would require like two - three
> weeks of work.
> I am sorry I didn't post any details, i was kind of busy because of a
> missed deadline :)
>
> The entity sync is kind of useless in this state imho.
> Reasons:
>
> 1) Does not do any fail over - if the sync process is interrupted for
> any reason - like network outage, or ofbiz restart, or system freeze
> it will be considered as a success sync and won't continue from the
> point it has failed. Could be fixed though :)
>
> 2) Pretty slow. The entities are transfered over the network using RMI.
> For transferring 5000 product records ( along with their relations -
> product_price, product_assoc, etc )
> it took over 2 hours using pretty good hard and 50 mbits network
> connection.
>
> 3) The example of entities to sync in the sync howto notes will never
> work. There will be records with duplicate PKs :)
>
> The solution for those issues imho are:
> 1) Refactor a bit the entity sync code so the client remembers the last
> successful sync state. This would fix the fail over issues. currently
> the server ( MCS ) remembers the
> sync status for each client.
>
> 2) Don't use RMI :) It would be much faster if the entities that are
> going to be transfered in order to get synced are serialized to XML for
> example and
> transferred over the network in a text format. RMI is easy to use but
> has bad performance. Just to give you an example:
> about 5000 products ( which is probably about 20 000 entities in total
> to pull ) takes 2 hours on a 50 mbits connection. What if we had a
> pretty huge supermarket
> which has 100 000 sales per day ( this means 100 000 sales orders ,
> average 50 order_items per order, 100 000 payments, 100 000 invoices,
> 100 000 invoice items ).
> It would require an entire night to sync - if we are lucky :)
>
> 3) I can provide the complete list of tables needed to get synced for a
> POS solution, including fix of the issue with the duplicate PKs :)
>
> However for our current project we've used triggers which create a
> transaction log for push on the POS databases and a transaction log for
> pull on the MCS instance.
> Pulling those 5000 products along with all the related records takes
> about 20 secs on the same hardware. I've configured a crontab job which
> invoces the sync postgre stored procedures
> every 5 minutes and it works perfectly. A status table is maintained on
> the MCS so we can check regularly if a POS hasn't synced for let's say 1
> hour - this means either network outage or some other problem.
>
> Yes, I know this solution is DB dependent but if the ofbiz sync
> framework issues are fixed and we use the list of tables to sync i've
> managed to create something useful could come out of it :)
>
> So - i hope my e-mail answers your question, if you want to know more -
> don't hesitate :)
>
> Cheers,
> Deyan
>
> -----Original Message-----
> From: Jacques Le Roux <ja...@les7arts.com>
> Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
> To: Deyan Tsvetanov <de...@ittconsult.com>,
> user@ofbiz.apache.org
> Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
> error )
> Date: Mon, 26 Oct 2009 08:34:51 +0100
>
>
> Hi Deyan,
>
> Did you work on this ? I will need to use EntitySync soon and I know you
> are not the 1st to complain about it (Si Chen did for instance). On the
> other hand, some seems to have used it successfully...
>
> Thanks
>
> Jacques
> ----- Original Message -----
> From: Deyan Tsvetanov
> To: Jacques Le Roux ; user@ofbiz.apache.org
> Sent: Sunday, October 04, 2009 10:57 AM
> Subject: EntitySync issues & suggestions ( was EntitySync RMI
> error )
>
>
>
> Hi Jacques and list,
>
> After playing for almost 2 weeks with the entity sync feature
> I've come up with the conclusion that it still needs
> some work to get ready for production environment. Currently it
> is unreliable, and useless :(
>
> - There is a bug in the design. The MCS remembers the sync
> status and currently reached sync timestamp for each client.
> This is not a good approach imho. Why ?
> The client asks the server: have you got something for me ?
> Next: The MCS checks the last successful sync timestamp for that
> particular client and selects the entities that should be passed
> to the client. After successfully transferring the entities to
> the client over RMI the MCS assumes that the sync is successful
> and updates the last successful sync timestamp.
>
> The problem is that the client does not confirm the success of
> the operation. If for some reason the client is unable to update
> the entities ( either shutdown,
> power failure, DB error or whatever ) the MCS isn't aware of
> that and the next sync will not include the missed entities.
> There is no way to fix the skipped
> entities but re-creating the client's database from zero.
>
> That is the main issue actually. It could be easily solved
> however. The most easy ways are:
> 1) The client will remember the sync timestamp and will include
> it as an argument when asking the server for pull sync. This way
> the client will
> update the last successful sync timestamp after storing each
> entity. The server does not have to remember anything regarding
> the client's sync status.
>
> 2) After each sync session the server will wait for confirmation
> from the client before updating the last successful sync
> timestamp.
> If not updated than the sync process is considered to be
> unsuccessful and the timestamp will not be updated.
>
> I personally prefer approach 1) as in case of network error,
> power failure or whatever error the sync process will continue
> from the place it has stopped.
>
>
> - There are some swallowed exceptions. If a network problem
> occurs during the sync process the server's status remains "sync
> running" and the client is
> not able to sync anymore until the status is reset by an
> operator.
>
> - The last issue is the performance. Using RMI is an easy
> approach from developer's perspective but it is very very slow.
> I had a database with 5000 products. The initial sync took about
> 2 hours. The hardware is pretty good, the network connection
> between the MCS and the client ofbiz
> is about 50 mbits. What if we have 100 000 sales per day ? The
> sync process will take probably the entire night :)
> Here I would suggest transferring GenericEntity instances over
> RMI to be removed. Instead a regular XML could be used. Either
> over RMI or SOAP - does not matter.
>
> So in general the picture would be:
>
> 1) Server does not remember any timestamp for any client.
> 2) Client calls the pull entity sync RMI ( or soap ) method and
> passes the entity sync ID and the timestamp of the last
> successful sync as arguments. Including the timezone of
> course :)
> 3) The server generates an XML file for all the entities in that
> entity sync group using the timestamp provided by the client.
> 4) Client starts updating the entities described in the received
> XML line by line. After each entity is updated successfully the
> timestamp is also updated ( on the client side - the
> server does not remember any client timestamps ).
> 5) If a failure occurs in the middle of the update process what
> we'll have is a sync process completed to like 50%. With the
> correct timestamp. When the next sync
> job starts the client will use the timestamp and the server will
> generate a new XML from the place the sync has failed. Of course
> an optimization could be used -
> the XML will be stored in a new db table so the client can
> continue storing the entities without asking the server to
> generate something he has already done.
>
> Some improvements that could be also useful:
> 1) The client ofbiz instances have sequence-id-prefix. So when
> the server is accepting push syncs it could also check the
> prefixes of the PKs and disallow entities without the correct
> prefix in the primary key. That's just a cheap insurance against
> errors :) We don't want store A to make sales on behalf of store
> B just because somebody made an error while configuring the
> pos ofbiz instance.
>
> 2) A time synchronization mechanism could be also implemented.
> In general when syncing there is a requirement that the server's
> and client's clocks are in sync ( ntp ). We don't want
> to set the system clock in ofbiz, but we could implement a check
> if the clocks are in sync and refuse to start the
> synchronization in certain cases.
>
> 3) A new column could be added to all the entities:
> isFromEntitySync : flag. This way we could avoid cases in which
> the client is trying to push records to the server which were
> actually pulled by the server. Yes, I know the entity's created
> timestamp should help us avoid such cases, but when the sync
> process is interrupted things like that could happen.
>
> Otherwise for our current project we've implemented a DB level
> synchronization - using triggers for INSERT, UPDATE and DELETE
> and a stored procedure which syncs the clients with a MCS.
> It works much ( tens of times ) faster than ofbiz entity sync .
> But it is DB dependent. So after getting comments and
> suggestions to this e-mail I would like to invest some time in
> fixing all those issues with the ofbiz entity sync and switch to
> it when it's ready :)
>
> Thanks for your time,
> Deyan
>
> -----Original Message-----
> From: Jacques Le Roux <ja...@les7arts.com>
> Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sun, 27 Sep 2009 17:02:57 +0200
>
>
> Hi Deyan,
>
> It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to
> put it, I
> will eventually take care to put it under FAQ...
>
> TIA
>
> Jacques
>
> From: "Deyan Tsvetanov" <de...@ittconsult.com>
> >I found an issue though:
> >
> > If the connection gets dropped during sync :
> > - the client ( POS ) prints connection reset by peer
> > - the server ( MCS ) sync status remains to running.
> > - the next sync does not start because MCS complains there is another
> > sync running already.
> >
> > "Reset run status" from webtools -> entity sync status
> > helps.
> >
> >
> > I'll investigate further and log a bug, although the solution seems
> > pretty simple - looks like a swallowed IOException,
> > which should be handled by re-setting the sync status.
> >
> > CHeers,
> > DEyan
> >
> > -----Original Message-----
> > From: Deyan Tsvetanov <de...@ittconsult.com>
> > Reply-to: user@ofbiz.apache.org
> > To: user@ofbiz.apache.org
> > Subject: Re: EntitySync RMI error
> > Date: Sat, 26 Sep 2009 11:22:55 +0300
> >
> >
> > Got it !
> >
> > Seems that the previous exception was coming from the MCS ( server ) .
> > When I stopped the server I started getting the exception I kind of like
> > more :)
> >
> >
> > Message: Exception calling remote pull and report EntitySync service
> > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> > ricServiceException: RMI Error (Connection refused to host:
> > 192.168.1.100; nested exception is:
> > java.net.ConnectException: Connection refused: connect)
> >
> > So the problem was at the server side - RMIIF env var in startofbiz.sh
> > wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> > hostname .
> >
> > The misleading things for me was that:
> > 1) no any sign in the MCS logs that somebody is trying to connect and
> > the MCS itself can not connect to its own RMI registry.
> > 2) no any difference in the exceptions text on the POS ( client ) side:
> > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> > remote one - can not connect to 127.0.0.1 )
> > look the same :)
> >
> > So issue is solved,
> > sorry for bothering :)
> >
> > -- deyan
> >
> > -----Original Message-----
> > From: Deyan Tsvetanov <de...@ittconsult.com>
> > Reply-to: user@ofbiz.apache.org
> > To: user@ofbiz.apache.org
> > Subject: Re: EntitySync RMI error
> > Date: Sat, 26 Sep 2009 09:52:55 +0300
> >
> >
> > During startup I get:
> >
> > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> > AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> > [main-rmi=rmi://127.0.0.
> > 1:1099/RMIDispatcher,
> > main-http=http://127.0.0.1:8080/webtools/control/httpService,
> > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> > 0.0.1:8080/webtools/control/httpService]
> >
> > entity-sync-rmi seems to be ok ...
> >
> >
> >
> >
> > -----Original Message-----
> > From: Deyan Tsvetanov <de...@ittconsult.com>
> > Reply-to: user@ofbiz.apache.org
> > To: user@ofbiz.apache.org
> > Subject: EntitySync RMI error
> > Date: Sat, 26 Sep 2009 09:22:23 +0300
> >
> >
> > Hi guys,
> >
> > I'm trying to configure RMI entity sync. I'm following
> > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
> >
> > What I've done so far:
> >
> > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
> >
> > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> > ( as per the example ).
> >
> > 3) I've imported the entity sync groups, SandJobs, etc.
> >
> > However when the sync starts ( on the POS instance ) I get the following
> > error:
> >
> > Exception calling remote pull and report EntitySync service with name:
> > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> > nested exception is:
> > java.net.ConnectException: Connection refused: connect)
> > Exception: org.ofbiz.service.GenericServiceException
> > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> > nested exception is:
> > java.net.ConnectException: Connection refused: connect)
> > ---- cause
> > ---------------------------------------------------------------------
> > Exception: java.rmi.ConnectException
> > Message: Connection refused to host: 127.0.0.1; nested exception is:
> > java.net.ConnectException: Connection refused: connect
> > ---- cause
> > ---------------------------------------------------------------------
> > Exception: java.net.ConnectException
> > Message: Connection refused: connect
> > ---- stack trace
> > ---------------------------------------------------------------
> >
> >
> > It insists connecting to 127.0.0.1 no matter what I type in
> > serviceengine.xml.
> > Any help would be appreciated :)
> >
> > Thanks in advance,
> > Deyan
> >
>
>
>
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Jacques Le Roux <ja...@les7arts.com>.
Thanks for comment Deyan,
I will try Pankaj Jain's tips in a 1st time
Jacques
From: "Deyan Tsvetanov" <de...@ittconsult.com>
Hi Jacques,
no, I didn't work on fixing the entity sync issues.
Instead I did my own sync tool using db triggers and stored procedures.
The entity sync could be fixed, but it would require like two - three
weeks of work.
I am sorry I didn't post any details, i was kind of busy because of a
missed deadline :)
The entity sync is kind of useless in this state imho.
Reasons:
1) Does not do any fail over - if the sync process is interrupted for
any reason - like network outage, or ofbiz restart, or system freeze
it will be considered as a success sync and won't continue from the
point it has failed. Could be fixed though :)
2) Pretty slow. The entities are transfered over the network using RMI.
For transferring 5000 product records ( along with their relations -
product_price, product_assoc, etc )
it took over 2 hours using pretty good hard and 50 mbits network
connection.
3) The example of entities to sync in the sync howto notes will never
work. There will be records with duplicate PKs :)
The solution for those issues imho are:
1) Refactor a bit the entity sync code so the client remembers the last
successful sync state. This would fix the fail over issues. currently
the server ( MCS ) remembers the
sync status for each client.
2) Don't use RMI :) It would be much faster if the entities that are
going to be transfered in order to get synced are serialized to XML for
example and
transferred over the network in a text format. RMI is easy to use but
has bad performance. Just to give you an example:
about 5000 products ( which is probably about 20 000 entities in total
to pull ) takes 2 hours on a 50 mbits connection. What if we had a
pretty huge supermarket
which has 100 000 sales per day ( this means 100 000 sales orders ,
average 50 order_items per order, 100 000 payments, 100 000 invoices,
100 000 invoice items ).
It would require an entire night to sync - if we are lucky :)
3) I can provide the complete list of tables needed to get synced for a
POS solution, including fix of the issue with the duplicate PKs :)
However for our current project we've used triggers which create a
transaction log for push on the POS databases and a transaction log for
pull on the MCS instance.
Pulling those 5000 products along with all the related records takes
about 20 secs on the same hardware. I've configured a crontab job which
invoces the sync postgre stored procedures
every 5 minutes and it works perfectly. A status table is maintained on
the MCS so we can check regularly if a POS hasn't synced for let's say 1
hour - this means either network outage or some other problem.
Yes, I know this solution is DB dependent but if the ofbiz sync
framework issues are fixed and we use the list of tables to sync i've
managed to create something useful could come out of it :)
So - i hope my e-mail answers your question, if you want to know more -
don't hesitate :)
Cheers,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: Deyan Tsvetanov <de...@ittconsult.com>,
user@ofbiz.apache.org
Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
error )
Date: Mon, 26 Oct 2009 08:34:51 +0100
Hi Deyan,
Did you work on this ? I will need to use EntitySync soon and I know you
are not the 1st to complain about it (Si Chen did for instance). On the
other hand, some seems to have used it successfully...
Thanks
Jacques
----- Original Message -----
From: Deyan Tsvetanov
To: Jacques Le Roux ; user@ofbiz.apache.org
Sent: Sunday, October 04, 2009 10:57 AM
Subject: EntitySync issues & suggestions ( was EntitySync RMI
error )
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature
I've come up with the conclusion that it still needs
some work to get ready for production environment. Currently it
is unreliable, and useless :(
- There is a bug in the design. The MCS remembers the sync
status and currently reached sync timestamp for each client.
This is not a good approach imho. Why ?
The client asks the server: have you got something for me ?
Next: The MCS checks the last successful sync timestamp for that
particular client and selects the entities that should be passed
to the client. After successfully transferring the entities to
the client over RMI the MCS assumes that the sync is successful
and updates the last successful sync timestamp.
The problem is that the client does not confirm the success of
the operation. If for some reason the client is unable to update
the entities ( either shutdown,
power failure, DB error or whatever ) the MCS isn't aware of
that and the next sync will not include the missed entities.
There is no way to fix the skipped
entities but re-creating the client's database from zero.
That is the main issue actually. It could be easily solved
however. The most easy ways are:
1) The client will remember the sync timestamp and will include
it as an argument when asking the server for pull sync. This way
the client will
update the last successful sync timestamp after storing each
entity. The server does not have to remember anything regarding
the client's sync status.
2) After each sync session the server will wait for confirmation
from the client before updating the last successful sync
timestamp.
If not updated than the sync process is considered to be
unsuccessful and the timestamp will not be updated.
I personally prefer approach 1) as in case of network error,
power failure or whatever error the sync process will continue
from the place it has stopped.
- There are some swallowed exceptions. If a network problem
occurs during the sync process the server's status remains "sync
running" and the client is
not able to sync anymore until the status is reset by an
operator.
- The last issue is the performance. Using RMI is an easy
approach from developer's perspective but it is very very slow.
I had a database with 5000 products. The initial sync took about
2 hours. The hardware is pretty good, the network connection
between the MCS and the client ofbiz
is about 50 mbits. What if we have 100 000 sales per day ? The
sync process will take probably the entire night :)
Here I would suggest transferring GenericEntity instances over
RMI to be removed. Instead a regular XML could be used. Either
over RMI or SOAP - does not matter.
So in general the picture would be:
1) Server does not remember any timestamp for any client.
2) Client calls the pull entity sync RMI ( or soap ) method and
passes the entity sync ID and the timestamp of the last
successful sync as arguments. Including the timezone of
course :)
3) The server generates an XML file for all the entities in that
entity sync group using the timestamp provided by the client.
4) Client starts updating the entities described in the received
XML line by line. After each entity is updated successfully the
timestamp is also updated ( on the client side - the
server does not remember any client timestamps ).
5) If a failure occurs in the middle of the update process what
we'll have is a sync process completed to like 50%. With the
correct timestamp. When the next sync
job starts the client will use the timestamp and the server will
generate a new XML from the place the sync has failed. Of course
an optimization could be used -
the XML will be stored in a new db table so the client can
continue storing the entities without asking the server to
generate something he has already done.
Some improvements that could be also useful:
1) The client ofbiz instances have sequence-id-prefix. So when
the server is accepting push syncs it could also check the
prefixes of the PKs and disallow entities without the correct
prefix in the primary key. That's just a cheap insurance against
errors :) We don't want store A to make sales on behalf of store
B just because somebody made an error while configuring the
pos ofbiz instance.
2) A time synchronization mechanism could be also implemented.
In general when syncing there is a requirement that the server's
and client's clocks are in sync ( ntp ). We don't want
to set the system clock in ofbiz, but we could implement a check
if the clocks are in sync and refuse to start the
synchronization in certain cases.
3) A new column could be added to all the entities:
isFromEntitySync : flag. This way we could avoid cases in which
the client is trying to push records to the server which were
actually pulled by the server. Yes, I know the entity's created
timestamp should help us avoid such cases, but when the sync
process is interrupted things like that could happen.
Otherwise for our current project we've implemented a DB level
synchronization - using triggers for INSERT, UPDATE and DELETE
and a stored procedure which syncs the clients with a MCS.
It works much ( tens of times ) faster than ofbiz entity sync .
But it is DB dependent. So after getting comments and
suggestions to this e-mail I would like to invest some time in
fixing all those issues with the ofbiz entity sync and switch to
it when it's ready :)
Thanks for your time,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: user@ofbiz.apache.org
Subject: Re: EntitySync RMI error
Date: Sun, 27 Sep 2009 17:02:57 +0200
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to
put it, I
will eventually take care to put it under FAQ...
TIA
Jacques
From: "Deyan Tsvetanov" <de...@ittconsult.com>
>I found an issue though:
>
> If the connection gets dropped during sync :
> - the client ( POS ) prints connection reset by peer
> - the server ( MCS ) sync status remains to running.
> - the next sync does not start because MCS complains there is another
> sync running already.
>
> "Reset run status" from webtools -> entity sync status
> helps.
>
>
> I'll investigate further and log a bug, although the solution seems
> pretty simple - looks like a swallowed IOException,
> which should be handled by re-setting the sync status.
>
> CHeers,
> DEyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 11:22:55 +0300
>
>
> Got it !
>
> Seems that the previous exception was coming from the MCS ( server ) .
> When I stopped the server I started getting the exception I kind of like
> more :)
>
>
> Message: Exception calling remote pull and report EntitySync service
> with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> ricServiceException: RMI Error (Connection refused to host:
> 192.168.1.100; nested exception is:
> java.net.ConnectException: Connection refused: connect)
>
> So the problem was at the server side - RMIIF env var in startofbiz.sh
> wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> hostname .
>
> The misleading things for me was that:
> 1) no any sign in the MCS logs that somebody is trying to connect and
> the MCS itself can not connect to its own RMI registry.
> 2) no any difference in the exceptions text on the POS ( client ) side:
> the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> remote one - can not connect to 127.0.0.1 )
> look the same :)
>
> So issue is solved,
> sorry for bothering :)
>
> -- deyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:52:55 +0300
>
>
> During startup I get:
>
> 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> [main-rmi=rmi://127.0.0.
> 1:1099/RMIDispatcher,
> main-http=http://127.0.0.1:8080/webtools/control/httpService,
> entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> 0.0.1:8080/webtools/control/httpService]
>
> entity-sync-rmi seems to be ok ...
>
>
>
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:22:23 +0300
>
>
> Hi guys,
>
> I'm trying to configure RMI entity sync. I'm following
> http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> What I've done so far:
>
> 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>
> 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> ( as per the example ).
>
> 3) I've imported the entity sync groups, SandJobs, etc.
>
> However when the sync starts ( on the POS instance ) I get the following
> error:
>
> Exception calling remote pull and report EntitySync service with name:
> remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> Exception: org.ofbiz.service.GenericServiceException
> Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.rmi.ConnectException
> Message: Connection refused to host: 127.0.0.1; nested exception is:
> java.net.ConnectException: Connection refused: connect
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.net.ConnectException
> Message: Connection refused: connect
> ---- stack trace
> ---------------------------------------------------------------
>
>
> It insists connecting to 127.0.0.1 no matter what I type in
> serviceengine.xml.
> Any help would be appreciated :)
>
> Thanks in advance,
> Deyan
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Deyan Tsvetanov <de...@ittconsult.com>.
Hi Jacques,
no, I didn't work on fixing the entity sync issues.
Instead I did my own sync tool using db triggers and stored procedures.
The entity sync could be fixed, but it would require like two - three
weeks of work.
I am sorry I didn't post any details, i was kind of busy because of a
missed deadline :)
The entity sync is kind of useless in this state imho.
Reasons:
1) Does not do any fail over - if the sync process is interrupted for
any reason - like network outage, or ofbiz restart, or system freeze
it will be considered as a success sync and won't continue from the
point it has failed. Could be fixed though :)
2) Pretty slow. The entities are transfered over the network using RMI.
For transferring 5000 product records ( along with their relations -
product_price, product_assoc, etc )
it took over 2 hours using pretty good hard and 50 mbits network
connection.
3) The example of entities to sync in the sync howto notes will never
work. There will be records with duplicate PKs :)
The solution for those issues imho are:
1) Refactor a bit the entity sync code so the client remembers the last
successful sync state. This would fix the fail over issues. currently
the server ( MCS ) remembers the
sync status for each client.
2) Don't use RMI :) It would be much faster if the entities that are
going to be transfered in order to get synced are serialized to XML for
example and
transferred over the network in a text format. RMI is easy to use but
has bad performance. Just to give you an example:
about 5000 products ( which is probably about 20 000 entities in total
to pull ) takes 2 hours on a 50 mbits connection. What if we had a
pretty huge supermarket
which has 100 000 sales per day ( this means 100 000 sales orders ,
average 50 order_items per order, 100 000 payments, 100 000 invoices,
100 000 invoice items ).
It would require an entire night to sync - if we are lucky :)
3) I can provide the complete list of tables needed to get synced for a
POS solution, including fix of the issue with the duplicate PKs :)
However for our current project we've used triggers which create a
transaction log for push on the POS databases and a transaction log for
pull on the MCS instance.
Pulling those 5000 products along with all the related records takes
about 20 secs on the same hardware. I've configured a crontab job which
invoces the sync postgre stored procedures
every 5 minutes and it works perfectly. A status table is maintained on
the MCS so we can check regularly if a POS hasn't synced for let's say 1
hour - this means either network outage or some other problem.
Yes, I know this solution is DB dependent but if the ofbiz sync
framework issues are fixed and we use the list of tables to sync i've
managed to create something useful could come out of it :)
So - i hope my e-mail answers your question, if you want to know more -
don't hesitate :)
Cheers,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: Deyan Tsvetanov <de...@ittconsult.com>,
user@ofbiz.apache.org
Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI
error )
Date: Mon, 26 Oct 2009 08:34:51 +0100
Hi Deyan,
Did you work on this ? I will need to use EntitySync soon and I know you
are not the 1st to complain about it (Si Chen did for instance). On the
other hand, some seems to have used it successfully...
Thanks
Jacques
----- Original Message -----
From: Deyan Tsvetanov
To: Jacques Le Roux ; user@ofbiz.apache.org
Sent: Sunday, October 04, 2009 10:57 AM
Subject: EntitySync issues & suggestions ( was EntitySync RMI
error )
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature
I've come up with the conclusion that it still needs
some work to get ready for production environment. Currently it
is unreliable, and useless :(
- There is a bug in the design. The MCS remembers the sync
status and currently reached sync timestamp for each client.
This is not a good approach imho. Why ?
The client asks the server: have you got something for me ?
Next: The MCS checks the last successful sync timestamp for that
particular client and selects the entities that should be passed
to the client. After successfully transferring the entities to
the client over RMI the MCS assumes that the sync is successful
and updates the last successful sync timestamp.
The problem is that the client does not confirm the success of
the operation. If for some reason the client is unable to update
the entities ( either shutdown,
power failure, DB error or whatever ) the MCS isn't aware of
that and the next sync will not include the missed entities.
There is no way to fix the skipped
entities but re-creating the client's database from zero.
That is the main issue actually. It could be easily solved
however. The most easy ways are:
1) The client will remember the sync timestamp and will include
it as an argument when asking the server for pull sync. This way
the client will
update the last successful sync timestamp after storing each
entity. The server does not have to remember anything regarding
the client's sync status.
2) After each sync session the server will wait for confirmation
from the client before updating the last successful sync
timestamp.
If not updated than the sync process is considered to be
unsuccessful and the timestamp will not be updated.
I personally prefer approach 1) as in case of network error,
power failure or whatever error the sync process will continue
from the place it has stopped.
- There are some swallowed exceptions. If a network problem
occurs during the sync process the server's status remains "sync
running" and the client is
not able to sync anymore until the status is reset by an
operator.
- The last issue is the performance. Using RMI is an easy
approach from developer's perspective but it is very very slow.
I had a database with 5000 products. The initial sync took about
2 hours. The hardware is pretty good, the network connection
between the MCS and the client ofbiz
is about 50 mbits. What if we have 100 000 sales per day ? The
sync process will take probably the entire night :)
Here I would suggest transferring GenericEntity instances over
RMI to be removed. Instead a regular XML could be used. Either
over RMI or SOAP - does not matter.
So in general the picture would be:
1) Server does not remember any timestamp for any client.
2) Client calls the pull entity sync RMI ( or soap ) method and
passes the entity sync ID and the timestamp of the last
successful sync as arguments. Including the timezone of
course :)
3) The server generates an XML file for all the entities in that
entity sync group using the timestamp provided by the client.
4) Client starts updating the entities described in the received
XML line by line. After each entity is updated successfully the
timestamp is also updated ( on the client side - the
server does not remember any client timestamps ).
5) If a failure occurs in the middle of the update process what
we'll have is a sync process completed to like 50%. With the
correct timestamp. When the next sync
job starts the client will use the timestamp and the server will
generate a new XML from the place the sync has failed. Of course
an optimization could be used -
the XML will be stored in a new db table so the client can
continue storing the entities without asking the server to
generate something he has already done.
Some improvements that could be also useful:
1) The client ofbiz instances have sequence-id-prefix. So when
the server is accepting push syncs it could also check the
prefixes of the PKs and disallow entities without the correct
prefix in the primary key. That's just a cheap insurance against
errors :) We don't want store A to make sales on behalf of store
B just because somebody made an error while configuring the
pos ofbiz instance.
2) A time synchronization mechanism could be also implemented.
In general when syncing there is a requirement that the server's
and client's clocks are in sync ( ntp ). We don't want
to set the system clock in ofbiz, but we could implement a check
if the clocks are in sync and refuse to start the
synchronization in certain cases.
3) A new column could be added to all the entities:
isFromEntitySync : flag. This way we could avoid cases in which
the client is trying to push records to the server which were
actually pulled by the server. Yes, I know the entity's created
timestamp should help us avoid such cases, but when the sync
process is interrupted things like that could happen.
Otherwise for our current project we've implemented a DB level
synchronization - using triggers for INSERT, UPDATE and DELETE
and a stored procedure which syncs the clients with a MCS.
It works much ( tens of times ) faster than ofbiz entity sync .
But it is DB dependent. So after getting comments and
suggestions to this e-mail I would like to invest some time in
fixing all those issues with the ofbiz entity sync and switch to
it when it's ready :)
Thanks for your time,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: user@ofbiz.apache.org
Subject: Re: EntitySync RMI error
Date: Sun, 27 Sep 2009 17:02:57 +0200
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I
will eventually take care to put it under FAQ...
TIA
Jacques
From: "Deyan Tsvetanov" <de...@ittconsult.com>
>I found an issue though:
>
> If the connection gets dropped during sync :
> - the client ( POS ) prints connection reset by peer
> - the server ( MCS ) sync status remains to running.
> - the next sync does not start because MCS complains there is another
> sync running already.
>
> "Reset run status" from webtools -> entity sync status
> helps.
>
>
> I'll investigate further and log a bug, although the solution seems
> pretty simple - looks like a swallowed IOException,
> which should be handled by re-setting the sync status.
>
> CHeers,
> DEyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 11:22:55 +0300
>
>
> Got it !
>
> Seems that the previous exception was coming from the MCS ( server ) .
> When I stopped the server I started getting the exception I kind of like
> more :)
>
>
> Message: Exception calling remote pull and report EntitySync service
> with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> ricServiceException: RMI Error (Connection refused to host:
> 192.168.1.100; nested exception is:
> java.net.ConnectException: Connection refused: connect)
>
> So the problem was at the server side - RMIIF env var in startofbiz.sh
> wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> hostname .
>
> The misleading things for me was that:
> 1) no any sign in the MCS logs that somebody is trying to connect and
> the MCS itself can not connect to its own RMI registry.
> 2) no any difference in the exceptions text on the POS ( client ) side:
> the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> remote one - can not connect to 127.0.0.1 )
> look the same :)
>
> So issue is solved,
> sorry for bothering :)
>
> -- deyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:52:55 +0300
>
>
> During startup I get:
>
> 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> [main-rmi=rmi://127.0.0.
> 1:1099/RMIDispatcher,
> main-http=http://127.0.0.1:8080/webtools/control/httpService,
> entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> 0.0.1:8080/webtools/control/httpService]
>
> entity-sync-rmi seems to be ok ...
>
>
>
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:22:23 +0300
>
>
> Hi guys,
>
> I'm trying to configure RMI entity sync. I'm following
> http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> What I've done so far:
>
> 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>
> 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> ( as per the example ).
>
> 3) I've imported the entity sync groups, SandJobs, etc.
>
> However when the sync starts ( on the POS instance ) I get the following
> error:
>
> Exception calling remote pull and report EntitySync service with name:
> remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> Exception: org.ofbiz.service.GenericServiceException
> Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.rmi.ConnectException
> Message: Connection refused to host: 127.0.0.1; nested exception is:
> java.net.ConnectException: Connection refused: connect
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.net.ConnectException
> Message: Connection refused: connect
> ---- stack trace
> ---------------------------------------------------------------
>
>
> It insists connecting to 127.0.0.1 no matter what I type in
> serviceengine.xml.
> Any help would be appreciated :)
>
> Thanks in advance,
> Deyan
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi Deyan,
Did you work on this ? I will need to use EntitySync soon and I know you are not the 1st to complain about it (Si Chen did for instance). On the other hand, some seems to have used it successfully...
Thanks
Jacques
----- Original Message -----
From: Deyan Tsvetanov
To: Jacques Le Roux ; user@ofbiz.apache.org
Sent: Sunday, October 04, 2009 10:57 AM
Subject: EntitySync issues & suggestions ( was EntitySync RMI error )
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs
some work to get ready for production environment. Currently it is unreliable, and useless :(
- There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client.
This is not a good approach imho. Why ?
The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp.
The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown,
power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped
entities but re-creating the client's database from zero.
That is the main issue actually. It could be easily solved however. The most easy ways are:
1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will
update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status.
2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp.
If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated.
I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped.
- There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is
not able to sync anymore until the status is reset by an operator.
- The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow.
I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz
is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :)
Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter.
So in general the picture would be:
1) Server does not remember any timestamp for any client.
2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :)
3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client.
4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the
server does not remember any client timestamps ).
5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync
job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used -
the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done.
Some improvements that could be also useful:
1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct
prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the
pos ofbiz instance.
2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want
to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases.
3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were
actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen.
Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS.
It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in
fixing all those issues with the ofbiz entity sync and switch to it when it's ready :)
Thanks for your time,
Deyan
-----Original Message-----
From: Jacques Le Roux <ja...@les7arts.com>
Reply-to: "Jacques Le Roux" <ja...@les7arts.com>
To: user@ofbiz.apache.org
Subject: Re: EntitySync RMI error
Date: Sun, 27 Sep 2009 17:02:57 +0200
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I
will eventually take care to put it under FAQ...
TIA
Jacques
From: "Deyan Tsvetanov" <de...@ittconsult.com>
>I found an issue though:
>
> If the connection gets dropped during sync :
> - the client ( POS ) prints connection reset by peer
> - the server ( MCS ) sync status remains to running.
> - the next sync does not start because MCS complains there is another
> sync running already.
>
> "Reset run status" from webtools -> entity sync status
> helps.
>
>
> I'll investigate further and log a bug, although the solution seems
> pretty simple - looks like a swallowed IOException,
> which should be handled by re-setting the sync status.
>
> CHeers,
> DEyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 11:22:55 +0300
>
>
> Got it !
>
> Seems that the previous exception was coming from the MCS ( server ) .
> When I stopped the server I started getting the exception I kind of like
> more :)
>
>
> Message: Exception calling remote pull and report EntitySync service
> with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene
> ricServiceException: RMI Error (Connection refused to host:
> 192.168.1.100; nested exception is:
> java.net.ConnectException: Connection refused: connect)
>
> So the problem was at the server side - RMIIF env var in startofbiz.sh
> wasn't set , so ofbiz was trying to get RMI host IP by resolving the
> hostname .
>
> The misleading things for me was that:
> 1) no any sign in the MCS logs that somebody is trying to connect and
> the MCS itself can not connect to its own RMI registry.
> 2) no any difference in the exceptions text on the POS ( client ) side:
> the 2 exceptions ( local one - can not connect to 192.168.1.100 and the
> remote one - can not connect to 127.0.0.1 )
> look the same :)
>
> So issue is solved,
> sorry for bothering :)
>
> -- deyan
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: Re: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:52:55 +0300
>
>
> During startup I get:
>
> 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [
> AbstractEngine.java:73 :INFO ] Loaded Service Locations :
> [main-rmi=rmi://127.0.0.
> 1:1099/RMIDispatcher,
> main-http=http://127.0.0.1:8080/webtools/control/httpService,
> entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher,
> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService,
> rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127.
> 0.0.1:8080/webtools/control/httpService]
>
> entity-sync-rmi seems to be ok ...
>
>
>
>
> -----Original Message-----
> From: Deyan Tsvetanov <de...@ittconsult.com>
> Reply-to: user@ofbiz.apache.org
> To: user@ofbiz.apache.org
> Subject: EntitySync RMI error
> Date: Sat, 26 Sep 2009 09:22:23 +0300
>
>
> Hi guys,
>
> I'm trying to configure RMI entity sync. I'm following
> http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> What I've done so far:
>
> 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher
>
> 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1
> ( as per the example ).
>
> 3) I've imported the entity sync groups, SandJobs, etc.
>
> However when the sync starts ( on the POS instance ) I get the following
> error:
>
> Exception calling remote pull and report EntitySync service with name:
> remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic
> eException: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> Exception: org.ofbiz.service.GenericServiceException
> Message: RMI Invocation Error (Connection refused to host: 127.0.0.1;
> nested exception is:
> java.net.ConnectException: Connection refused: connect)
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.rmi.ConnectException
> Message: Connection refused to host: 127.0.0.1; nested exception is:
> java.net.ConnectException: Connection refused: connect
> ---- cause
> ---------------------------------------------------------------------
> Exception: java.net.ConnectException
> Message: Connection refused: connect
> ---- stack trace
> ---------------------------------------------------------------
>
>
> It insists connecting to 127.0.0.1 no matter what I type in
> serviceengine.xml.
> Any help would be appreciated :)
>
> Thanks in advance,
> Deyan
>
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Jacques Le Roux <ja...@les7arts.com>.
Thanks Pankaj,
I had not the time to read all your message, very useful I guess.
I will post all this (with Deyan's remark too) on Wiki after having done my own opinion...
Jacques
From: <Pa...@lntinfotech.com>
> Hi Deyan n others
>
> I have also worked & solved many sync problems in ofbiz in my application:
>
> 1) As you said while sync is in progress if server crashes, the status in
> EntitySync table gets in running state and we can't sync further until we
> manually change it even if we restart server it remains in running state
> To solve this problem: I created a new entity SyncStatus on client side
> to maintain the status of sync, in that ist status is Not Started, when
> service runEntitySync or runPullEntitySync starts I set the status of
> field as running and when service returns (whether fail or successful) I
> updated status to finish.
> I also added code, that if server restarts and if this field status is
> running change it to not started, By this I solved the above problem:
>
> *) If client crashes, then after restart, status in SyncStatus table will
> be Not Started.
> *) If server crashes, then service will be failed and status will be
> finished and no further problem.
>
> 2) There were other problems like if sync(push or pull) is running, and
> one other job of same service is also created then we get exception "An
> instance is already running" By using above status in SyncStatus table
> when A job of sync is running, I just cancelled out all other jobs of same
> type.
>
> 3) I also solved "Connection refused" problem by same method even I
> displayed message on screen rather than giving exception on console.
>
> 4) I did not get your point of updating time stamp, I did not get this
> problem because In push or pull sync the successful timestamp is updated
> on EntitySync table on client or MCS side only if sync is successful, if
> server crashes or sync fails this field is not updated, but yes there are
> some issues:
> *) If we have 2 hours of sync, n sync fails on last minute. then in next
> sync as sync successful status has not been revisesd, next sync will pick
> the status of previous one, as a solution I think we need to keep updating
> syn status in EntitySync table after particular intervals. but how to
> decide these intervals b'coz tables are linked with foreign key
> constriants.
> *) One problem with connection refused in push and pull case is that, In
> pull sync first connection is established then data is prepared for sync
> while in push sync first data is prepared and then conncetion is
> established. So in pull sync we get connection refused problem in earlier
> stage than push sync. We can say in push sync if there is no MCS (Or MCS
> IP address is wrong), Our code is doing efforts in preparing data that
> effort is of no use.
>
> 5) If we place order on POS side and try to perform push sync, some time I
> was getting random exceptions due to order header or Order Status table, I
> solved the problem.
> Explain: In placing order (or others) if you have used a database trigger
> (eecas) like if an entry is created on A table create an entry on B table.
> Now wile sync if MCS is also having same trigger then as a entry is
> created (by sync) on A table , trigger is fired and an entry for B is
> created. Now for one A, B will be having two entries that creates random
> exception
> So I want to ask, that mean for sync there should not be ant trigger on
> opposite side(means for push MCS and for pull POS), how we will maintain
> that ???
>
> I did not get your concept of using XML file inspite of RMI. Could you
> please give some details regarding that???? Even I agree with the fact
> that RMI takes long time.
>
>
> Regards:
>
> Pankaj Jain
>
>
>
> ______________________________________________________________________
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
Posted by Pa...@lntinfotech.com.
Hi Deyan n others
I have also worked & solved many sync problems in ofbiz in my application:
1) As you said while sync is in progress if server crashes, the status in
EntitySync table gets in running state and we can't sync further until we
manually change it even if we restart server it remains in running state
To solve this problem: I created a new entity SyncStatus on client side
to maintain the status of sync, in that ist status is Not Started, when
service runEntitySync or runPullEntitySync starts I set the status of
field as running and when service returns (whether fail or successful) I
updated status to finish.
I also added code, that if server restarts and if this field status is
running change it to not started, By this I solved the above problem:
*) If client crashes, then after restart, status in SyncStatus table will
be Not Started.
*) If server crashes, then service will be failed and status will be
finished and no further problem.
2) There were other problems like if sync(push or pull) is running, and
one other job of same service is also created then we get exception "An
instance is already running" By using above status in SyncStatus table
when A job of sync is running, I just cancelled out all other jobs of same
type.
3) I also solved "Connection refused" problem by same method even I
displayed message on screen rather than giving exception on console.
4) I did not get your point of updating time stamp, I did not get this
problem because In push or pull sync the successful timestamp is updated
on EntitySync table on client or MCS side only if sync is successful, if
server crashes or sync fails this field is not updated, but yes there are
some issues:
*) If we have 2 hours of sync, n sync fails on last minute. then in next
sync as sync successful status has not been revisesd, next sync will pick
the status of previous one, as a solution I think we need to keep updating
syn status in EntitySync table after particular intervals. but how to
decide these intervals b'coz tables are linked with foreign key
constriants.
*) One problem with connection refused in push and pull case is that, In
pull sync first connection is established then data is prepared for sync
while in push sync first data is prepared and then conncetion is
established. So in pull sync we get connection refused problem in earlier
stage than push sync. We can say in push sync if there is no MCS (Or MCS
IP address is wrong), Our code is doing efforts in preparing data that
effort is of no use.
5) If we place order on POS side and try to perform push sync, some time I
was getting random exceptions due to order header or Order Status table, I
solved the problem.
Explain: In placing order (or others) if you have used a database trigger
(eecas) like if an entry is created on A table create an entry on B table.
Now wile sync if MCS is also having same trigger then as a entry is
created (by sync) on A table , trigger is fired and an entry for B is
created. Now for one A, B will be having two entries that creates random
exception
So I want to ask, that mean for sync there should not be ant trigger on
opposite side(means for push MCS and for pull POS), how we will maintain
that ???
I did not get your concept of using XML file inspite of RMI. Could you
please give some details regarding that???? Even I agree with the fact
that RMI takes long time.
Regards:
Pankaj Jain
______________________________________________________________________