You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ojb-dev@db.apache.org by "Robert S. Sfeir" <ro...@codepuccino.com> on 2004/06/23 15:56:08 UTC
Thoughts on Locking issues
Hey guys,
Real quick before I disappear for the day...
I was discussing some ideas about hot to handle optimistic locking in post
1.0 with Brian, and here are my thoughts on a couple of possible solutions.
First idea:
To handle locking correctly across all DBs we have to rely on our own
mechanism, and I think that the best way to do this would be to have our own
table which stores the object id and its serial id in that table.
On object select, you'd update that table with the ids of the returned
objects. Now I know this can be a lot of overhead, so my suggestion is to
perhaps have something like a servlet running which can batch process the
ids after the select is done and data is returned to the user. Something
that would happen in the background. Chances are the user will not be fast
enough to update the objects to be affected by the small delay in updating
the ojb lock table. This servlet could also be multithreaded but I don't
think it's necessary.
Second thought to this idea is that instead of just locking objects on
select, we can provide a method which does a special select, a
getObjectsForEdit(), where when this method is called, that's the time that
you update objects to the lock table. This way when you do a 'normal'
select, you get the objects and nothing gets marked for lock, and we
minimize the overhead.
In either case, there should also be an option for automatic deprecation of
locks. This should be something set by the user. Say a user selects a
bunch of objects, if another user comes along and selects the objects and
updates them, we check with the table, if the last time these objects are >
then the pre determined timestamp, we override the lock, and allow the
update. If it's less we give the engineer a choice to figure out what to
do. These choices I think would be:
1- Throw an error and tell the user object is already locked for edit
2- Give them a method to try a gain without having to write too much
code, something like tryPersistAgain(numTimesToTry) so we can write a while
loop which would try to persist it the specified number of times, then fail
again.
3- ignore lock completely and update, in which case we update our lock
table so the next user has the same choices.
Second idea, which I don't think is as reliable to code:
Do same as first but without tables, store objectid and serialid in a
Hashtable as objects get selected. This will probably too memory intensive,
however we could have a process which removes objectids from the Hashtable
after a certain period of time so we can keep memory to a minimum. I have
code which I used for cached table data in one of my apps which does this.
In my case it works great, but I don't know how much we can rely on it for
this purpose. Here is my code for it:
(In either case we can use this code to either persist the objects in the
Hashtable to the DB or just keep it there, code needs slight mod to retrofit
into the suggestion above.)
/**
* Stores data that was selected from the database into id/name pairs in a
Hashtable for later fast retrievals. This
* is meant to get single values quickly. If you need lists you need to
look at RexListFunctions. Generally use this
* method when you need to display just the result of a name based on an
ID. For example if you have a userID
* displayed in a form, instead of the ID look up the name of the user
based on the ID in the hashtable
*
* @param category
* @param name
* @param id
*/
public static void cacheNameAndID( final String category, final String
name, final int id )
{
Hashtable map = _getCategoryMap( category + ".nameByID" );
map.put( new Integer( id ), name );
map = _getCategoryMap( category + ".idByName" );
map.put( name, new Integer( id ) );
}
private static Hashtable _getCategoryMap( final String category )
{
Hashtable map = ( Hashtable ) categoryMap.get( category );
if( map == null )
{
map = ( Hashtable ) categoryMap.get( category );
if( map == null )
{
map = new Hashtable();
categoryMap.put( category, map );
}
}
return map;
}
/**
* Gets the Name of a value in a category from a hashtable using the ID.
*
* @param category the hashtable name. .nameByID will be appended to
this.
* @param id the ID of the name you're looking for.
*
* @return the value of the ID; the name.
*/
public static String getNameByID( final String category, final int id )
{
final Hashtable map = _getCategoryMap( category + ".nameByID" );
if( map.get( new Integer( id ) ) == null )
{
return "";
}
else
{
return ( String ) map.get( new Integer( id ) );
}
}
/**
* Gets the ID of a name in a category from a hashtable based on a name.
*
* @param category the hashtable name. .idByName will be appended to
this.
* @param name the name who's ID you're looking for
*
* @return the value of the name; the ID.
*/
public static int getIDByName( final String category, final String name )
{
final Hashtable map = _getCategoryMap( category + ".idByName" );
return ( ( Integer ) map.get( name ) ).intValue();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: Thoughts on Locking issues
Posted by Thomas Mahler <th...@web.de>.
Hi Robert,
Robert S. Sfeir wrote:
> Hey guys,
>
> Real quick before I disappear for the day...
>
> I was discussing some ideas about hot to handle optimistic locking in post
> 1.0 with Brian, and here are my thoughts on a couple of possible solutions.
>
> First idea:
>
> To handle locking correctly across all DBs we have to rely on our own
> mechanism, and I think that the best way to do this would be to have our own
> table which stores the object id and its serial id in that table.
>
Maybe I missed something. What is wrong with the current OL implementation?
The current issue with the nanoseconds problems is caused by bugs in the
Oracle driver.
There is also a simple patch as a workaround.
Requiring extra tables for any features gives bad press from the users.
It will also be used by the propaganda machinery of our toughest
comepetitor as an argument against us.
So I'm really against introducing such a table.
> On object select, you'd update that table with the ids of the returned
> objects. Now I know this can be a lot of overhead, so my suggestion is to
> perhaps have something like a servlet running which can batch process the
> ids after the select is done and data is returned to the user. Something
> that would happen in the background. Chances are the user will not be fast
> enough to update the objects to be affected by the small delay in updating
> the ojb lock table. This servlet could also be multithreaded but I don't
> think it's necessary.
>
For the ODMG pessimistic locking i did the following:
I'm using a Map that hold all the locks.
the user can choose which Map Implementation to use:
1. a persistent map based on a database table
2. a Servlet based remote Map. The Servlet does not provide any lock
persistence, but hold the actual lockmap in memory.
Works like a charm and is MUCH faster then the DB based lockmap.
I'd prefer to use this approach if it is really required to change the
OL locking stuff.
just my 2c
Thomas
> Second thought to this idea is that instead of just locking objects on
> select, we can provide a method which does a special select, a
> getObjectsForEdit(), where when this method is called, that's the time that
> you update objects to the lock table. This way when you do a 'normal'
> select, you get the objects and nothing gets marked for lock, and we
> minimize the overhead.
>
> In either case, there should also be an option for automatic deprecation of
> locks. This should be something set by the user. Say a user selects a
> bunch of objects, if another user comes along and selects the objects and
> updates them, we check with the table, if the last time these objects are >
> then the pre determined timestamp, we override the lock, and allow the
> update. If it's less we give the engineer a choice to figure out what to
> do. These choices I think would be:
> 1- Throw an error and tell the user object is already locked for edit
> 2- Give them a method to try a gain without having to write too much
> code, something like tryPersistAgain(numTimesToTry) so we can write a while
> loop which would try to persist it the specified number of times, then fail
> again.
> 3- ignore lock completely and update, in which case we update our lock
> table so the next user has the same choices.
>
>
> Second idea, which I don't think is as reliable to code:
>
> Do same as first but without tables, store objectid and serialid in a
> Hashtable as objects get selected. This will probably too memory intensive,
> however we could have a process which removes objectids from the Hashtable
> after a certain period of time so we can keep memory to a minimum. I have
> code which I used for cached table data in one of my apps which does this.
> In my case it works great, but I don't know how much we can rely on it for
> this purpose. Here is my code for it:
>
> (In either case we can use this code to either persist the objects in the
> Hashtable to the DB or just keep it there, code needs slight mod to retrofit
> into the suggestion above.)
>
> /**
> * Stores data that was selected from the database into id/name pairs in a
> Hashtable for later fast retrievals. This
> * is meant to get single values quickly. If you need lists you need to
> look at RexListFunctions. Generally use this
> * method when you need to display just the result of a name based on an
> ID. For example if you have a userID
> * displayed in a form, instead of the ID look up the name of the user
> based on the ID in the hashtable
> *
> * @param category
> * @param name
> * @param id
> */
> public static void cacheNameAndID( final String category, final String
> name, final int id )
> {
> Hashtable map = _getCategoryMap( category + ".nameByID" );
> map.put( new Integer( id ), name );
> map = _getCategoryMap( category + ".idByName" );
> map.put( name, new Integer( id ) );
> }
>
> private static Hashtable _getCategoryMap( final String category )
> {
> Hashtable map = ( Hashtable ) categoryMap.get( category );
> if( map == null )
> {
> map = ( Hashtable ) categoryMap.get( category );
> if( map == null )
> {
> map = new Hashtable();
> categoryMap.put( category, map );
> }
> }
> return map;
> }
>
> /**
> * Gets the Name of a value in a category from a hashtable using the ID.
> *
> * @param category the hashtable name. .nameByID will be appended to
> this.
> * @param id the ID of the name you're looking for.
> *
> * @return the value of the ID; the name.
> */
> public static String getNameByID( final String category, final int id )
> {
> final Hashtable map = _getCategoryMap( category + ".nameByID" );
> if( map.get( new Integer( id ) ) == null )
> {
> return "";
> }
> else
> {
> return ( String ) map.get( new Integer( id ) );
> }
> }
>
> /**
> * Gets the ID of a name in a category from a hashtable based on a name.
> *
> * @param category the hashtable name. .idByName will be appended to
> this.
> * @param name the name who's ID you're looking for
> *
> * @return the value of the name; the ID.
> */
> public static int getIDByName( final String category, final String name )
> {
> final Hashtable map = _getCategoryMap( category + ".idByName" );
> return ( ( Integer ) map.get( name ) ).intValue();
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: Thoughts on Locking issues
Posted by "Robert S. Sfeir" <ro...@codepuccino.com>.
Yeah there is a problem in a multithreaded approach, and until recently
Postgresql kept locking on some of the tests and I could do it consistently.
Put a bigger load on OJB like the multithreaded test but with thousands of
records, and it will fail not matter what you do. While this might not seem
like an issue because as Martin mentioned the test itself is trying 100
times and is that enough etc... I could at one point reproduce the error
consistently, and I am concerned that in a multithreaded, high load
environment the current methodology is just not good enough.
Of course that would be up to me to write such a test to prove it, and most
likely write at least part of the new impl. The second solution is just
fine with me, and perhaps in any case could be offered as an option.
R
On 6/26/04 7:06 AM, "Jakob Braeuchi" <jb...@gmx.ch> wrote:
> hi robert,
>
>
> Robert S. Sfeir wrote:
>> Hey guys,
>>
>> Real quick before I disappear for the day...
>>
>> I was discussing some ideas about hot to handle optimistic locking in post
>> 1.0 with Brian, and here are my thoughts on a couple of possible solutions.
>>
>> First idea:
>>
>> To handle locking correctly across all DBs we have to rely on our own
>> mechanism, and I think that the best way to do this would be to have our own
>> table which stores the object id and its serial id in that table.
>
> this sounds like pessimistic locking to me ! and it requires an additional
> table. do we really have a problem with the current implementation of
> optimistic
> locking ?
>
>>
>> On object select, you'd update that table with the ids of the returned
>> objects. Now I know this can be a lot of overhead, so my suggestion is to
>> perhaps have something like a servlet running which can batch process the
>> ids after the select is done and data is returned to the user. Something
>> that would happen in the background. Chances are the user will not be fast
>> enough to update the objects to be affected by the small delay in updating
>> the ojb lock table. This servlet could also be multithreaded but I don't
>> think it's necessary.
>
> imo this is too much overhead. i'd prefer the second solution (if any) where
> you
> lock object you want to modify. but again here: it's pessimistic and needs a
> special table.
>
>>
>> Second thought to this idea is that instead of just locking objects on
>> select, we can provide a method which does a special select, a
>> getObjectsForEdit(), where when this method is called, that's the time that
>> you update objects to the lock table. This way when you do a 'normal'
>> select, you get the objects and nothing gets marked for lock, and we
>> minimize the overhead.
>>
>> In either case, there should also be an option for automatic deprecation of
>> locks. This should be something set by the user. Say a user selects a
>> bunch of objects, if another user comes along and selects the objects and
>> updates them, we check with the table, if the last time these objects are >
>> then the pre determined timestamp, we override the lock, and allow the
>> update. If it's less we give the engineer a choice to figure out what to
>> do. These choices I think would be:
>> 1- Throw an error and tell the user object is already locked for edit
>> 2- Give them a method to try a gain without having to write too much
>> code, something like tryPersistAgain(numTimesToTry) so we can write a while
>> loop which would try to persist it the specified number of times, then fail
>> again.
>> 3- ignore lock completely and update, in which case we update our lock
>> table so the next user has the same choices.
>>
>>
>> Second idea, which I don't think is as reliable to code:
>>
>> Do same as first but without tables, store objectid and serialid in a
>> Hashtable as objects get selected. This will probably too memory intensive,
>
> you could store only the objects you intend to edit to reduce memory usage.
> but
> this solution works only for one vm.
>
> jakob
>
>> however we could have a process which removes objectids from the Hashtable
>> after a certain period of time so we can keep memory to a minimum. I have
>> code which I used for cached table data in one of my apps which does this.
>> In my case it works great, but I don't know how much we can rely on it for
>> this purpose. Here is my code for it:
>>
>> (In either case we can use this code to either persist the objects in the
>> Hashtable to the DB or just keep it there, code needs slight mod to retrofit
>> into the suggestion above.)
>>
>> /**
>> * Stores data that was selected from the database into id/name pairs in a
>> Hashtable for later fast retrievals. This
>> * is meant to get single values quickly. If you need lists you need to
>> look at RexListFunctions. Generally use this
>> * method when you need to display just the result of a name based on an
>> ID. For example if you have a userID
>> * displayed in a form, instead of the ID look up the name of the user
>> based on the ID in the hashtable
>> *
>> * @param category
>> * @param name
>> * @param id
>> */
>> public static void cacheNameAndID( final String category, final String
>> name, final int id )
>> {
>> Hashtable map = _getCategoryMap( category + ".nameByID" );
>> map.put( new Integer( id ), name );
>> map = _getCategoryMap( category + ".idByName" );
>> map.put( name, new Integer( id ) );
>> }
>>
>> private static Hashtable _getCategoryMap( final String category )
>> {
>> Hashtable map = ( Hashtable ) categoryMap.get( category );
>> if( map == null )
>> {
>> map = ( Hashtable ) categoryMap.get( category );
>> if( map == null )
>> {
>> map = new Hashtable();
>> categoryMap.put( category, map );
>> }
>> }
>> return map;
>> }
>>
>> /**
>> * Gets the Name of a value in a category from a hashtable using the ID.
>> *
>> * @param category the hashtable name. .nameByID will be appended to
>> this.
>> * @param id the ID of the name you're looking for.
>> *
>> * @return the value of the ID; the name.
>> */
>> public static String getNameByID( final String category, final int id )
>> {
>> final Hashtable map = _getCategoryMap( category + ".nameByID" );
>> if( map.get( new Integer( id ) ) == null )
>> {
>> return "";
>> }
>> else
>> {
>> return ( String ) map.get( new Integer( id ) );
>> }
>> }
>>
>> /**
>> * Gets the ID of a name in a category from a hashtable based on a name.
>> *
>> * @param category the hashtable name. .idByName will be appended to
>> this.
>> * @param name the name who's ID you're looking for
>> *
>> * @return the value of the name; the ID.
>> */
>> public static int getIDByName( final String category, final String name )
>> {
>> final Hashtable map = _getCategoryMap( category + ".idByName" );
>> return ( ( Integer ) map.get( name ) ).intValue();
>> }
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: Thoughts on Locking issues
Posted by Jakob Braeuchi <jb...@gmx.ch>.
hi robert,
Robert S. Sfeir wrote:
> Hey guys,
>
> Real quick before I disappear for the day...
>
> I was discussing some ideas about hot to handle optimistic locking in post
> 1.0 with Brian, and here are my thoughts on a couple of possible solutions.
>
> First idea:
>
> To handle locking correctly across all DBs we have to rely on our own
> mechanism, and I think that the best way to do this would be to have our own
> table which stores the object id and its serial id in that table.
this sounds like pessimistic locking to me ! and it requires an additional
table. do we really have a problem with the current implementation of optimistic
locking ?
>
> On object select, you'd update that table with the ids of the returned
> objects. Now I know this can be a lot of overhead, so my suggestion is to
> perhaps have something like a servlet running which can batch process the
> ids after the select is done and data is returned to the user. Something
> that would happen in the background. Chances are the user will not be fast
> enough to update the objects to be affected by the small delay in updating
> the ojb lock table. This servlet could also be multithreaded but I don't
> think it's necessary.
imo this is too much overhead. i'd prefer the second solution (if any) where you
lock object you want to modify. but again here: it's pessimistic and needs a
special table.
>
> Second thought to this idea is that instead of just locking objects on
> select, we can provide a method which does a special select, a
> getObjectsForEdit(), where when this method is called, that's the time that
> you update objects to the lock table. This way when you do a 'normal'
> select, you get the objects and nothing gets marked for lock, and we
> minimize the overhead.
>
> In either case, there should also be an option for automatic deprecation of
> locks. This should be something set by the user. Say a user selects a
> bunch of objects, if another user comes along and selects the objects and
> updates them, we check with the table, if the last time these objects are >
> then the pre determined timestamp, we override the lock, and allow the
> update. If it's less we give the engineer a choice to figure out what to
> do. These choices I think would be:
> 1- Throw an error and tell the user object is already locked for edit
> 2- Give them a method to try a gain without having to write too much
> code, something like tryPersistAgain(numTimesToTry) so we can write a while
> loop which would try to persist it the specified number of times, then fail
> again.
> 3- ignore lock completely and update, in which case we update our lock
> table so the next user has the same choices.
>
>
> Second idea, which I don't think is as reliable to code:
>
> Do same as first but without tables, store objectid and serialid in a
> Hashtable as objects get selected. This will probably too memory intensive,
you could store only the objects you intend to edit to reduce memory usage. but
this solution works only for one vm.
jakob
> however we could have a process which removes objectids from the Hashtable
> after a certain period of time so we can keep memory to a minimum. I have
> code which I used for cached table data in one of my apps which does this.
> In my case it works great, but I don't know how much we can rely on it for
> this purpose. Here is my code for it:
>
> (In either case we can use this code to either persist the objects in the
> Hashtable to the DB or just keep it there, code needs slight mod to retrofit
> into the suggestion above.)
>
> /**
> * Stores data that was selected from the database into id/name pairs in a
> Hashtable for later fast retrievals. This
> * is meant to get single values quickly. If you need lists you need to
> look at RexListFunctions. Generally use this
> * method when you need to display just the result of a name based on an
> ID. For example if you have a userID
> * displayed in a form, instead of the ID look up the name of the user
> based on the ID in the hashtable
> *
> * @param category
> * @param name
> * @param id
> */
> public static void cacheNameAndID( final String category, final String
> name, final int id )
> {
> Hashtable map = _getCategoryMap( category + ".nameByID" );
> map.put( new Integer( id ), name );
> map = _getCategoryMap( category + ".idByName" );
> map.put( name, new Integer( id ) );
> }
>
> private static Hashtable _getCategoryMap( final String category )
> {
> Hashtable map = ( Hashtable ) categoryMap.get( category );
> if( map == null )
> {
> map = ( Hashtable ) categoryMap.get( category );
> if( map == null )
> {
> map = new Hashtable();
> categoryMap.put( category, map );
> }
> }
> return map;
> }
>
> /**
> * Gets the Name of a value in a category from a hashtable using the ID.
> *
> * @param category the hashtable name. .nameByID will be appended to
> this.
> * @param id the ID of the name you're looking for.
> *
> * @return the value of the ID; the name.
> */
> public static String getNameByID( final String category, final int id )
> {
> final Hashtable map = _getCategoryMap( category + ".nameByID" );
> if( map.get( new Integer( id ) ) == null )
> {
> return "";
> }
> else
> {
> return ( String ) map.get( new Integer( id ) );
> }
> }
>
> /**
> * Gets the ID of a name in a category from a hashtable based on a name.
> *
> * @param category the hashtable name. .idByName will be appended to
> this.
> * @param name the name who's ID you're looking for
> *
> * @return the value of the name; the ID.
> */
> public static int getIDByName( final String category, final String name )
> {
> final Hashtable map = _getCategoryMap( category + ".idByName" );
> return ( ( Integer ) map.get( name ) ).intValue();
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org