generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

Hello all,

I have a situation where i frequently have to merge data from different
databases. Uptill now every database has it's own sequence. That means that
i cannot merge the data blindly because two records with the same id might
not be the same record. I am thinking of switching from int id's to varchar
UID's. Does ojb support such mechanism in an offline manner, so that the two
servers do not need to communicate with each other. If all id's are globally
unique i can blindly merge data without the danger of overwriting stuff. If
ojb does not offer this out of the box then does anybody now a way to
generate UID's across systems. Within one system I use
java.rmi.server.UID().toString()
but i dont know how unique that is across different VM's.

Cheers,
Dennis

Re: generate global uid's

[Armin Waibel <ar...@apache.org>]

Hi Dennis,

Dennis Bekkering wrote:
> Sorry, nope! At the beginning of OJB some people plan to support beside
>> RDBMS and other persistence datastores, but it was too complex to
>> realize with a few OJB committer.
> 
> 
> Thanks for clearing out, where exactly lies the complexity, the quering i
> solved my own project.

E.g. to support ODBMS we have to adapt the PB-api and rewrite internal 
classes because many of them are tied to RDBMS.


> I have the biggest problem with activationdepth, 
> with
> OJB there isn't such concept (aka lazy loading). Dunno of that is 
> db4objects
> specific or also SODA. With how many are you working currently on OJB?
>

About four active committer.

regards,
Armin

> Cheers,
> Dennis
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

Sorry, nope! At the beginning of OJB some people plan to support beside
> RDBMS and other persistence datastores, but it was too complex to
> realize with a few OJB committer.


Thanks for clearing out, where exactly lies the complexity, the quering i
solved my own project. I have the biggest problem with activationdepth, with
OJB there isn't such concept (aka lazy loading). Dunno of that is db4objects
specific or also SODA. With how many are you working currently on OJB?

Cheers,
Dennis

Re: generate global uid's

[Armin Waibel <ar...@apache.org>]

Hi Denins,

Dennis Bekkering wrote:
> Hi,
> 
> I found out that the select IN problem had to do with different charsets on
> the column level, since my id's where varchars this problem occured 
> suddenly
> , i checked all column charset but couldn't find a difference between them.
> The deeper i get into mysql the more i find out why not everybody likes 
> this
> database that much. It is very hard to determine what is wrong and why. 
> What
> are other developers favorite database either open source or not? Are there
> databases that have meaningfull error messages anyway?
> 
> Regarding the GUID's and collections i tried another appraoch then i first
> suggested. I added an xtra guid field to every table, now tables have a
> local int id and a global varchar one. I did this to be able to move on but
> it is better anyway i think. For collections (either 1:N or M:N) i only 
> send
> the GUID's of the members in that collections. If the new records are send
> before that then the collections can be assambled thereafter without having
> to fall back on sql. The cache doesn't have to be cleared also. The last 2
> years i tried several approaches but i think this one will actually work.
> For me It is one of the hardest problems to find a solution for. Anyway i
> had to browse the OJB source and that was fun, i discovered source files
> regarding SODA. What is the status of this?

The SODA support is deprecated and will be removed.
http://db.apache.org/ojb/status.html

Think the current implementation is rudimentary.


> I made a conceptual interface
> implementation for SODA, the same interface that i use to query OJB, but if
> OJB already supports SODA i would be crazy to build it myself. Can i for
> instance use db4objects with OJB?

Sorry, nope! At the beginning of OJB some people plan to support beside 
RDBMS and other persistence datastores, but it was too complex to 
realize with a few OJB committer.

regards,
Armin

> 
> Cheers,
> Dennis
> 
> 
> 2006/5/13, Dennis Bekkering <dj...@gmail.com>:
>>
>> All,
>>
>> I converted all my int id's and refs to varchar id's and ref's, i
>> offcourse updated the repository file also. Now it seems that OJB expects
>> id's to be of type int in 1:N and M:N relations. I get errors regarding
>> select where id IN (?,?,?,?,?). I did not really looked in the source 
>> yet at
>> what is happening, so my questioin is if it is at all possible to use
>> varchars in 1:N and M:N relations. I can not post the log because i 
>> have the
>> new stuff at home only right now and am at work (on saterday?, yes i am a
>> crazy nerd ;-). If it should be possible i will dive into it deeper 
>> and try
>> to find what i am doing wrong.
>>
>> Cheers,
>> Dennis
>>
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

Hi,

I found out that the select IN problem had to do with different charsets on
the column level, since my id's where varchars this problem occured suddenly
, i checked all column charset but couldn't find a difference between them.
The deeper i get into mysql the more i find out why not everybody likes this
database that much. It is very hard to determine what is wrong and why. What
are other developers favorite database either open source or not? Are there
databases that have meaningfull error messages anyway?

Regarding the GUID's and collections i tried another appraoch then i first
suggested. I added an xtra guid field to every table, now tables have a
local int id and a global varchar one. I did this to be able to move on but
it is better anyway i think. For collections (either 1:N or M:N) i only send
the GUID's of the members in that collections. If the new records are send
before that then the collections can be assambled thereafter without having
to fall back on sql. The cache doesn't have to be cleared also. The last 2
years i tried several approaches but i think this one will actually work.
For me It is one of the hardest problems to find a solution for. Anyway i
had to browse the OJB source and that was fun, i discovered source files
regarding SODA. What is the status of this? I made a conceptual interface
implementation for SODA, the same interface that i use to query OJB, but if
OJB already supports SODA i would be crazy to build it myself. Can i for
instance use db4objects with OJB?

Cheers,
Dennis


2006/5/13, Dennis Bekkering <dj...@gmail.com>:
>
> All,
>
> I converted all my int id's and refs to varchar id's and ref's, i
> offcourse updated the repository file also. Now it seems that OJB expects
> id's to be of type int in 1:N and M:N relations. I get errors regarding
> select where id IN (?,?,?,?,?). I did not really looked in the source yet at
> what is happening, so my questioin is if it is at all possible to use
> varchars in 1:N and M:N relations. I can not post the log because i have the
> new stuff at home only right now and am at work (on saterday?, yes i am a
> crazy nerd ;-). If it should be possible i will dive into it deeper and try
> to find what i am doing wrong.
>
> Cheers,
> Dennis
>



-- 
mvg,
Dennis

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

All,

I converted all my int id's and refs to varchar id's and ref's, i offcourse
updated the repository file also. Now it seems that OJB expects id's to be
of type int in 1:N and M:N relations. I get errors regarding select where id
IN (?,?,?,?,?). I did not really looked in the source yet at what is
happening, so my questioin is if it is at all possible to use varchars in
1:N and M:N relations. I can not post the log because i have the new stuff
at home only right now and am at work (on saterday?, yes i am a crazy nerd
;-). If it should be possible i will dive into it deeper and try to find
what i am doing wrong.

Cheers,
Dennis

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

Armin,

If you need an additional field in indirection table you can't use
> non-decomposed m:n relation.


ok

To make this possible (OJB populate an additional timestamp column in
> indirection table on insert) with non-decomposed m:n relation you have
> to modify/extend OJB. If each indirection table has a additional column
> TIME_STAMP you can modify the creation of m:n insert statement. In this
> case you have to implement your own SqlGenerator class (extend existing
> default and override #getInsertMNStatement) and MtoNBroker class
> (override insert methods)... this will be complicated.


If there is one method that eventually inserts i cannot imagine that it is
really complicated. I will have a look at it. I dont need to update values
because the creationdate is only relevant on inserts.

I don't know an easy solution for your problem.
> If the timeframe between two synchronization calls is greater then
> seconds and you don't need millisecond precision of timestamp values you
> can try to update the indirection table entries after OJB insert/update
> the entries. You can use plain SQL or you can define a persistence
> capable object for each indirection table and after insert/update you
> request all rows with empty timestamp column, set the timestamp and
> update the indirection table objects.


Let me tell you a little more about the plan and the context. I store all my
OJB repository equivalent meta data in a model that is stored in the
database itself and is just another bunch of tables that is also backed by
OJB. I generate OJB repository files from it (and also hibernate). When i am
finished working at home, where i have no internet, i export my database and
import it at work. When somebody else at work also updates the meta model in
the meantime i am in trouble. So it should right now basically be a meta
data versioning system. Time frames between synq actions will span hours if
not days. I plan to use plain SQL, really simple just loop all tables and
records and check their last modified date (lastmodified i can control from
my code). It is the M2M stuff that only needs creationdate because they
represent structure not data. The M2M records are present or they are
removed, nothing else.

I think you provided me with the answer, all i need is a creation stamp in
the composed m2m relations. I will give that a shot and let you know how it
works out.

Cheers and a nice weekend to all,
Dennis

Re: generate global uid's

[Armin Waibel <ar...@apache.org>]

Hi Dennis,

Dennis Bekkering wrote:
> Hello all,
> 
> I am almost there with regards to asynq data merging based upon GUID. 
> What i
> am looking for now is a way to have a creationdate field in
> indirectiontables. I am not sure how to do this since it is all handled by
> OJB.

If you need an additional field in indirection table you can't use 
non-decomposed m:n relation.

> 
> The case is like this, a local server has records that are not present in
> the main server. It is either a new local record or a deleted global one. I
> can discover what it is if i know the last synqdate of the localserver. If
> the lastsynq date is after the creationdate of the local record then the
> global record is deleted and the local record should also be deleted, if 
> the
> lastsynq is before the creationdate then the local record is new and should
> be send to the main server. I can control every record's creationdate 
> except
> for the indirecttable ones.

To make this possible (OJB populate an additional timestamp column in 
indirection table on insert) with non-decomposed m:n relation you have 
to modify/extend OJB. If each indirection table has a additional column 
TIME_STAMP you can modify the creation of m:n insert statement. In this 
case you have to implement your own SqlGenerator class (extend existing 
default and override #getInsertMNStatement) and MtoNBroker class 
(override insert methods)... this will be complicated.

I don't know an easy solution for your problem.
If the timeframe between two synchronization calls is greater then 
seconds and you don't need millisecond precision of timestamp values you 
can try to update the indirection table entries after OJB insert/update 
the entries. You can use plain SQL or you can define a persistence 
capable object for each indirection table and after insert/update you 
request all rows with empty timestamp column, set the timestamp and 
update the indirection table objects.

regards,
Armin



> 
> regards,
> Dennis
> 
> 
> 
> 2006/4/25, Armin Waibel <ar...@apache.org>:
>>
>> Hi Dennis,
>>
>> to use OJB's auto-increment feature (automatic assignment of PK's) you
>> have to implement a SequenceManager based on class GUIDFactory.
>>
>> Simply extend AbstractSequenceManager and override
>> public Object getUniqueValue(...)
>>
>> Implement
>> protected int getUniqueId(...)
>> and throw exception within this method (see 
>> SequenceManagerMSSQLGuidImpl).
>>
>> regards,
>> Armin
>>
>>
>> Dennis Bekkering wrote:
>> > great, thanks!
>> >
>> > 2006/4/24, Thomas Mahler <th...@web.de>:
>> >> Hi Dennis,
>> >>
>> >> org.apache.ojb.broker.util.GUIDFactory produces GUIDs that are unique
>> >> across different VMs.
>> >>
>> >> cheers,
>> >> Thomas
>> >>
>> >> Dennis Bekkering wrote:
>> >>> Hello all,
>> >>>
>> >>> I have a situation where i frequently have to merge data from
>> different
>> >>> databases. Uptill now every database has it's own sequence. That 
>> means
>> >> that
>> >>> i cannot merge the data blindly because two records with the same id
>> >> might
>> >>> not be the same record. I am thinking of switching from int id's to
>> >> varchar
>> >>> UID's. Does ojb support such mechanism in an offline manner, so that
>> the
>> >> two
>> >>> servers do not need to communicate with each other. If all id's are
>> >> globally
>> >>> unique i can blindly merge data without the danger of overwriting
>> stuff.
>> >> If
>> >>> ojb does not offer this out of the box then does anybody now a way to
>> >>> generate UID's across systems. Within one system I use
>> >>> java.rmi.server.UID().toString()
>> >>> but i dont know how unique that is across different VM's.
>> >>>
>> >>> Cheers,
>> >>> Dennis
>> >>>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> >> For additional commands, e-mail: ojb-user-help@db.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > mvg,
>> > Dennis
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

Hello all,

I am almost there with regards to asynq data merging based upon GUID. What i
am looking for now is a way to have a creationdate field in
indirectiontables. I am not sure how to do this since it is all handled by
OJB.

The case is like this, a local server has records that are not present in
the main server. It is either a new local record or a deleted global one. I
can discover what it is if i know the last synqdate of the localserver. If
the lastsynq date is after the creationdate of the local record then the
global record is deleted and the local record should also be deleted, if the
lastsynq is before the creationdate then the local record is new and should
be send to the main server. I can control every record's creationdate except
for the indirecttable ones.

regards,
Dennis



2006/4/25, Armin Waibel <ar...@apache.org>:
>
> Hi Dennis,
>
> to use OJB's auto-increment feature (automatic assignment of PK's) you
> have to implement a SequenceManager based on class GUIDFactory.
>
> Simply extend AbstractSequenceManager and override
> public Object getUniqueValue(...)
>
> Implement
> protected int getUniqueId(...)
> and throw exception within this method (see SequenceManagerMSSQLGuidImpl).
>
> regards,
> Armin
>
>
> Dennis Bekkering wrote:
> > great, thanks!
> >
> > 2006/4/24, Thomas Mahler <th...@web.de>:
> >> Hi Dennis,
> >>
> >> org.apache.ojb.broker.util.GUIDFactory produces GUIDs that are unique
> >> across different VMs.
> >>
> >> cheers,
> >> Thomas
> >>
> >> Dennis Bekkering wrote:
> >>> Hello all,
> >>>
> >>> I have a situation where i frequently have to merge data from
> different
> >>> databases. Uptill now every database has it's own sequence. That means
> >> that
> >>> i cannot merge the data blindly because two records with the same id
> >> might
> >>> not be the same record. I am thinking of switching from int id's to
> >> varchar
> >>> UID's. Does ojb support such mechanism in an offline manner, so that
> the
> >> two
> >>> servers do not need to communicate with each other. If all id's are
> >> globally
> >>> unique i can blindly merge data without the danger of overwriting
> stuff.
> >> If
> >>> ojb does not offer this out of the box then does anybody now a way to
> >>> generate UID's across systems. Within one system I use
> >>> java.rmi.server.UID().toString()
> >>> but i dont know how unique that is across different VM's.
> >>>
> >>> Cheers,
> >>> Dennis
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> >> For additional commands, e-mail: ojb-user-help@db.apache.org
> >>
> >>
> >
> >
> > --
> > mvg,
> > Dennis
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>


-- 
mvg,
Dennis

Re: generate global uid's

[Armin Waibel <ar...@apache.org>]

Hi Dennis,

to use OJB's auto-increment feature (automatic assignment of PK's) you 
have to implement a SequenceManager based on class GUIDFactory.

Simply extend AbstractSequenceManager and override
public Object getUniqueValue(...)

Implement
protected int getUniqueId(...)
and throw exception within this method (see SequenceManagerMSSQLGuidImpl).

regards,
Armin


Dennis Bekkering wrote:
> great, thanks!
> 
> 2006/4/24, Thomas Mahler <th...@web.de>:
>> Hi Dennis,
>>
>> org.apache.ojb.broker.util.GUIDFactory produces GUIDs that are unique
>> across different VMs.
>>
>> cheers,
>> Thomas
>>
>> Dennis Bekkering wrote:
>>> Hello all,
>>>
>>> I have a situation where i frequently have to merge data from different
>>> databases. Uptill now every database has it's own sequence. That means
>> that
>>> i cannot merge the data blindly because two records with the same id
>> might
>>> not be the same record. I am thinking of switching from int id's to
>> varchar
>>> UID's. Does ojb support such mechanism in an offline manner, so that the
>> two
>>> servers do not need to communicate with each other. If all id's are
>> globally
>>> unique i can blindly merge data without the danger of overwriting stuff.
>> If
>>> ojb does not offer this out of the box then does anybody now a way to
>>> generate UID's across systems. Within one system I use
>>> java.rmi.server.UID().toString()
>>> but i dont know how unique that is across different VM's.
>>>
>>> Cheers,
>>> Dennis
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
> 
> 
> --
> mvg,
> Dennis
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org

Re: generate global uid's

[Dennis Bekkering <dj...@gmail.com>]

great, thanks!

2006/4/24, Thomas Mahler <th...@web.de>:
>
> Hi Dennis,
>
> org.apache.ojb.broker.util.GUIDFactory produces GUIDs that are unique
> across different VMs.
>
> cheers,
> Thomas
>
> Dennis Bekkering wrote:
> > Hello all,
> >
> > I have a situation where i frequently have to merge data from different
> > databases. Uptill now every database has it's own sequence. That means
> that
> > i cannot merge the data blindly because two records with the same id
> might
> > not be the same record. I am thinking of switching from int id's to
> varchar
> > UID's. Does ojb support such mechanism in an offline manner, so that the
> two
> > servers do not need to communicate with each other. If all id's are
> globally
> > unique i can blindly merge data without the danger of overwriting stuff.
> If
> > ojb does not offer this out of the box then does anybody now a way to
> > generate UID's across systems. Within one system I use
> > java.rmi.server.UID().toString()
> > but i dont know how unique that is across different VM's.
> >
> > Cheers,
> > Dennis
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>


--
mvg,
Dennis

Re: generate global uid's

[Thomas Mahler <th...@web.de>]

Hi Dennis,

org.apache.ojb.broker.util.GUIDFactory produces GUIDs that are unique 
across different VMs.

cheers,
Thomas

Dennis Bekkering wrote:
> Hello all,
> 
> I have a situation where i frequently have to merge data from different
> databases. Uptill now every database has it's own sequence. That means that
> i cannot merge the data blindly because two records with the same id might
> not be the same record. I am thinking of switching from int id's to varchar
> UID's. Does ojb support such mechanism in an offline manner, so that the two
> servers do not need to communicate with each other. If all id's are globally
> unique i can blindly merge data without the danger of overwriting stuff. If
> ojb does not offer this out of the box then does anybody now a way to
> generate UID's across systems. Within one system I use
> java.rmi.server.UID().toString()
> but i dont know how unique that is across different VM's.
> 
> Cheers,
> Dennis
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org