You are viewing a plain text version of this content. The canonical link for it is here.
Posted to torque-dev@db.apache.org by Thomas Fischer <fi...@seitenbau.net> on 2005/07/29 15:07:40 UTC
silent db hit for associated objects
Hi,
If one has an association between objects by foreign key, (e.g. book ->
author association in the tutorial), Torque generates methods to get the
associated objects (e.g. Book.getAuthor() and Author.getBooks()). If the
associated object has not yet been retrieved, these methods silently ask
the database for the objects, using a new connection from the database
pool.
This is not at all what one wants if one wants to use transactions in an
application, because one wants to do all reads in the same transaction. I
therefore plan to do the following
1) add a method getXXX(boolean silentDbFetch), which only hits the database
silently if silentDbFetch is set to true. If set to false and the
associated object(s) have not been retrieved yet, null is returned, even
though there might be associated objects in the database.
2 ) add a method getXXX(connection) for the n->1 direction, whcih uses the
provided connection if a db hit is necessary (in the book example, this
would be book.getAuthor(connection) (author.getBooks() was already present)
)
3) add a generator property torque.silentDbFetch, which is true by default.
If set to false, the getXXX() methods without arguments will not access the
db silently, but return null if the associated objects has not been read
yet. Also, the method does not throw a Torque exception any more (which is
the main reason why using these methods is more convenient than
getXXX(boolean silentDbFetch) with silentDbFetch set to false).
Any comments, objections, better ideas, better name for the silentDbFetch
parameter? In particular, I am not so sure if 3) is useful for the general
public, but then it helps to make sure that one does not read from the
database out of neglect if one does not want to do so.
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Fischer <tf...@apache.org>.
On Tue, 16 Aug 2005, Thomas Vandahl wrote:
> Thomas Fischer wrote:
>
>> Some time after that I want to access the data I have stored, e.g. because
>> the user has submitted a command in an HTML form to change the data. The
>> only "official" (i.e. automatically generated) way to get the cached Books
>> from an author is to use author.getBooks(). But I care about the data being
>> consistent, so in no circumstances do I want to access the database,
>> because this would break transaction safety and lead to inconsistent data
>> if soneone elase has changed the data in the menatime. Especially, if, by
>> some mistake, I did forget to read the books of an author, I want to know
>> it. A logical way of knowing is that author.getBooks() returns null.
>
> If you are /sure/ that you're using the very same instance of your author to
> modify something, there is every reason to believe that your books fetched
> before are still there. I see your point for debugging purposes. But that
> should not happen in production code.
Yes. But to be sure that it does not happen, one needs to get a clean
error each time one tries to make it happen. If one does not get a clean
error, but a silent db hit instead, the chance that someone notices during
development that something is wrong is very small. I have seen it.
>> I am personally using xslt, but that does not play any role. The problem is
>> that I have no idea to provide the methods when managers are turned on. The
>> implementation without the connection asks the manager to provide the
>> object, and there is no way I have found to tell the manager "get that
>> object, but if it is not there, use the provided connection". So I did not
>> implement it in objectWithManager.vm. Do you have any idea of how to solve
>> this one ?
>
> Not without bigger changes in the AbstractBaseManager. I will have a look at
> this. I guess the real fun starts if I want to cache the whole collection of
> books at once (using MethodResultCache for example, not to speak of
> invalidation).
I must admit that I have only the slightest idea what the Managers do (I
was never tempted to use them, given that I cannot imagine they work very
well if you have different JVMs writing to the same database), so if you
think this could be useful and would take a look at it, I would be very
glad.
>
>> If you transfer it to java, yes. I would explicitly use book.getAuthor()
>> in the presentation layer.
>
> I see your point. With Velocity it makes no difference:
>
> Author a = AuthorPeer.retrieveByPK(id, con);
> context.put("author", a);
>
> List books = a .getBooks(con);
> context.put("books", books);
It would make a difference if yoor object tree gets more complex. Just
imagine you would have a list of authors, each of
which has a list of books. So using caching, it would be just two lines
List authors = AuthorPeer.doSelectJoinBooks(criteria, connection);
context.put("authors", authors);
Without caching, that would be horrible (i.e. you would have to
reconstruct the structure cahcing already provides.) This will get
worse if your object tree gets more and more complicated.
>
>> I would not think so. If one knows that the property torque.objectIsCaching
>> exists, in the generator, and one decides not to set it to false, one
>> explicitly wants to have the caching. And if one has it, is it not legimite
>> to say "I only want to use the cached data, and not hit the database"
>
> Still it looks a bit clumsy to me.
>
Yes, it certainly is, but I did not see a better way to give the user that
choice. Another way would have been to add another method getXXX(boolean
silentDbHit), but I already had Henning's flames on that, and he is right
(we already have too many methods in the object classes).
> Bye, Thomas.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: torque-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Fischer <fi...@seitenbau.net>.
Hi Thomas,
though I have no experience with the Cache, the code looks good to me.
Thomas
Thomas Vandahl <th...@tewisoft.de> schrieb am 08.09.2005 16:31:16:
> Thomas Vandahl wrote:
> > Not without bigger changes in the AbstractBaseManager. I will have a
> > look at this. I guess the real fun starts if I want to cache the whole
> > collection of books at once (using MethodResultCache for example, not
to
> > speak of invalidation).
>
> Replying to myself is fun...
>
> Well, I finally came up with an attempt for a solution. It was less work
> than I expected, but I guess we are only halfway through. It passes the
> runtime test, however.
>
> It looks like this:
> ---8<---
> public Author getAuthor(Connection connection)
> throws TorqueException
> {
> if (aAuthor == null && (this.authorId != 0))
> {
> aAuthor =
> AuthorManager.getCachedInstance(SimpleKey.keyFor(this.authorId));
> if (aAuthor == null)
> {
> aAuthor =
> AuthorPeer.retrieveByPK(SimpleKey.keyFor(this.authorId), connection);
> AuthorManager.putInstance(aAuthor);
> }
> }
> return aAuthor;
> }
> ---8<---
>
> While I was at it I corrected a few problems PMD complained about, but
> there is still room for improvement... :-)
>
> Any comments are welcome.
>
> Bye, Thomas.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: torque-dev-help@db.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Vandahl <th...@tewisoft.de>.
Thomas Vandahl wrote:
> Not without bigger changes in the AbstractBaseManager. I will have a
> look at this. I guess the real fun starts if I want to cache the whole
> collection of books at once (using MethodResultCache for example, not to
> speak of invalidation).
Replying to myself is fun...
Well, I finally came up with an attempt for a solution. It was less work
than I expected, but I guess we are only halfway through. It passes the
runtime test, however.
It looks like this:
---8<---
public Author getAuthor(Connection connection)
throws TorqueException
{
if (aAuthor == null && (this.authorId != 0))
{
aAuthor =
AuthorManager.getCachedInstance(SimpleKey.keyFor(this.authorId));
if (aAuthor == null)
{
aAuthor =
AuthorPeer.retrieveByPK(SimpleKey.keyFor(this.authorId), connection);
AuthorManager.putInstance(aAuthor);
}
}
return aAuthor;
}
---8<---
While I was at it I corrected a few problems PMD complained about, but
there is still room for improvement... :-)
Any comments are welcome.
Bye, Thomas.
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Vandahl <th...@tewisoft.de>.
Thomas Fischer wrote:
> Some time after that I want to access the data I have stored, e.g.
> because the user has submitted a command in an HTML form to change the
> data. The only "official" (i.e. automatically generated) way to get the
> cached Books from an author is to use author.getBooks(). But I care
> about the data being consistent, so in no circumstances do I want to
> access the database, because this would break transaction safety and
> lead to inconsistent data if soneone elase has changed the data in the
> menatime. Especially, if, by some mistake, I did forget to read the
> books of an author, I want to know it. A logical way of knowing is that
> author.getBooks() returns null.
If you are /sure/ that you're using the very same instance of your
author to modify something, there is every reason to believe that your
books fetched before are still there. I see your point for debugging
purposes. But that should not happen in production code.
> I am personally using xslt, but that does not play any role. The problem
> is that I have no idea to provide the methods when managers are turned
> on. The implementation without the connection asks the manager to
> provide the object, and there is no way I have found to tell the manager
> "get that object, but if it is not there, use the provided connection".
> So I did not implement it in objectWithManager.vm. Do you have any idea
> of how to solve this one ?
Not without bigger changes in the AbstractBaseManager. I will have a
look at this. I guess the real fun starts if I want to cache the whole
collection of books at once (using MethodResultCache for example, not to
speak of invalidation).
> If you transfer it to java, yes. I would explicitly use book.getAuthor()
> in the presentation layer.
I see your point. With Velocity it makes no difference:
Author a = AuthorPeer.retrieveByPK(id, con);
context.put("author", a);
List books = a .getBooks(con);
context.put("books", books);
> I would not think so. If one knows that the property
> torque.objectIsCaching exists, in the generator, and one decides not to
> set it to false, one explicitly wants to have the caching. And if one
> has it, is it not legimite to say "I only want to use the cached data,
> and not hit the database"
Still it looks a bit clumsy to me.
Bye, Thomas.
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Fischer <tf...@apache.org>.
Hi Thomas,
thanks for your answer.
On Tue, 16 Aug 2005, Thomas Vandahl wrote:
> Thomas Fischer wrote:
>> On Tue, 2 Aug 2005, Henning P. Schmiedehausen wrote:
>>> Thomas Fischer <fi...@seitenbau.net> writes:
> [...]
>> Hm, personally, I always use torque.objectIsCaching = true. The reason is
>> as follows: In most applications, I need transaction safety. Then, I like
>> to separate the application into a data accesss layer and the presentation
>> layer. But the presentation layer should be able to retrieve associated
>> objects, in the bookstore example this would be e.g. to be able to ask a
>> book for its author. If I want to make this transaction safe and would have
>> set torque.objectIsCaching to false, then I would have to pass the database
>> connection to the presentation layer, which is not what a presentation
>> layer is about.
>
> If I use the getXXX methods without a connection parameter, the expected
> behaviour from my point of view *is* a DB-hit.
No it is not, if you accept that one may want to cache a whole tree of
objects. Imagine the following: I have loaded an Author ant his
corresponding books in a transaction-safe manner, and stored the author
somewhere (e.g in a Http session). Knowing that the associated books will
also be stored with the author, I do not store them separately. (ok, for a
single Author, it would be feasible to store his books separately, but
consider a lost of authors and their associated books). Then I commit the
transaction, and I know that all data I have in memory is consistent.
Some time after that I want to access the data I have stored, e.g. because
the user has submitted a command in an HTML form to change the data. The
only "official" (i.e. automatically generated) way to get the cached Books
from an author is to use author.getBooks(). But I care about the data
being consistent, so in no circumstances do I want to access the database,
because this would break transaction safety and lead to inconsistent data
if soneone elase has changed the data in the menatime. Especially, if, by
some mistake, I did forget to read the books of an author, I want to know
it. A logical way of knowing is that author.getBooks() returns null.
So in this scenario, the automatic db hit is very contraproductive, and
there is no easy means to prevent it.
> It is more questionable
> whether the implicit caching of the collections or related objects is a good
> idea at all. It hurts especially when using managers.
Then you should consider to set the generator property
torque.objectIsCaching to false.
>
> I guess that getXXX(Connection con) would be a good extension anyway, because
> it makes the set of methods more logical. This way, you can always collect
> your data in a transaction and provide it to the presentation layer in the
> context separately (we talk Velocity here, don't we? :-)
I am personally using xslt, but that does not play any role. The problem
is that I have no idea to provide the methods when managers are turned on.
The implementation without the connection asks the manager to provide the
object, and there is no way I have found to tell the manager "get that
object, but if it is not there, use the provided connection". So I did not
implement it in objectWithManager.vm. Do you have any idea of how to solve
this one ?
>
>>>> 3) add a generator property torque.silentDbFetch, which is true by
>>>> default.
>>>> If set to false, the getXXX() methods without arguments will not access
>>>> the
>>>> db silently, but return null if the associated objects has not been read
>>>> yet. Also, the method does not throw a Torque exception any more (which
>>>> is
>>>> the main reason why using these methods is more convenient than
>>>> getXXX(boolean silentDbFetch) with silentDbFetch set to false).
>
> Personally, I don't see that you gain much with this. As I understand it,
> this means that you need to prefetch all related objects in the business code
> in a transaction like
>
> Book b = BookPeer.retrieveByPK(id, con);
> b.getAuthor(con);
> context.put("book", b);
>
> and then use
>
> $book.Author
>
> in the presentation layer. Is that the case?
If you transfer it to java, yes. I would explicitly use book.getAuthor()
in the presentation layer.
> I consider this difficult to
> read. You need to know that there is something cached under the hood to
> understand what's going on here...
I would not think so. If one knows that the property
torque.objectIsCaching exists, in the generator, and one decides not to
set it to false, one explicitly wants to have the caching. And if one has
it, is it not legimite to say "I only want to use the cached data, and not
hit the database"
>
> I kindly ask you to sleep over it again.
One does not have to use it. It is not the default
behaviour we are talking about (this will stay as it is, of course) but a
decision you can take if you know what you are doing.
I would agree that if you have uncritical data you would not use it, but
for critical data it is an absolute nightmare to have uncontrolled access
to a database outside of transaction. Imaginge having to find an error in
an application which has a problem like that ! Huh, it makes me shudder...
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Vandahl <th...@tewisoft.de>.
Thomas Fischer wrote:
> On Tue, 2 Aug 2005, Henning P. Schmiedehausen wrote:
>> Thomas Fischer <fi...@seitenbau.net> writes:
[...]
> Hm, personally, I always use torque.objectIsCaching = true. The reason
> is as follows: In most applications, I need transaction safety. Then, I
> like to separate the application into a data accesss layer and the
> presentation layer. But the presentation layer should be able to
> retrieve associated objects, in the bookstore example this would be e.g.
> to be able to ask a book for its author. If I want to make this
> transaction safe and would have set torque.objectIsCaching to false,
> then I would have to pass the database connection to the presentation
> layer, which is not what a presentation layer is about.
If I use the getXXX methods without a connection parameter, the expected
behaviour from my point of view *is* a DB-hit. It is more questionable
whether the implicit caching of the collections or related objects is a
good idea at all. It hurts especially when using managers.
I guess that getXXX(Connection con) would be a good extension anyway,
because it makes the set of methods more logical. This way, you can
always collect your data in a transaction and provide it to the
presentation layer in the context separately (we talk Velocity here,
don't we? :-)
>>> 3) add a generator property torque.silentDbFetch, which is true by
>>> default.
>>> If set to false, the getXXX() methods without arguments will not
>>> access the
>>> db silently, but return null if the associated objects has not been read
>>> yet. Also, the method does not throw a Torque exception any more
>>> (which is
>>> the main reason why using these methods is more convenient than
>>> getXXX(boolean silentDbFetch) with silentDbFetch set to false).
Personally, I don't see that you gain much with this. As I understand
it, this means that you need to prefetch all related objects in the
business code in a transaction like
Book b = BookPeer.retrieveByPK(id, con);
b.getAuthor(con);
context.put("book", b);
and then use
$book.Author
in the presentation layer. Is that the case? I consider this difficult
to read. You need to know that there is something cached under the hood
to understand what's going on here...
I kindly ask you to sleep over it again.
Bye, Thomas.
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by Thomas Fischer <tf...@apache.org>.
On Tue, 2 Aug 2005, Henning P. Schmiedehausen wrote:
> Thomas Fischer <fi...@seitenbau.net> writes:
>
>> Hi,
>
>> If one has an association between objects by foreign key, (e.g. book ->
>> author association in the tutorial), Torque generates methods to get the
>> associated objects (e.g. Book.getAuthor() and Author.getBooks()). If the
>> associated object has not yet been retrieved, these methods silently ask
>> the database for the objects, using a new connection from the database
>> pool.
>> This is not at all what one wants if one wants to use transactions in an
>> application, because one wants to do all reads in the same transaction. I
>> therefore plan to do the following
>> 1) add a method getXXX(boolean silentDbFetch), which only hits the database
>> silently if silentDbFetch is set to true. If set to false and the
>> associated object(s) have not been retrieved yet, null is returned, even
>> though there might be associated objects in the database.
>
> -0 The classes are already too overloaded IMHO.
Ok, good point. One gets carried away sometimes. I will remove it again
before I commit it (Personally I do not use it anyway)
>
>> 2 ) add a method getXXX(connection) for the n->1 direction, whcih uses the
>> provided connection if a db hit is necessary (in the book example, this
>> would be book.getAuthor(connection) (author.getBooks() was already present)
>
> +0; this is better, but still not really good. The question is: Does it
> hurt? The database hit is read-only, so a rollback would only be
> necessary if you have some trigger in the database which does writing
> changes on a read-access. In the worst case you will get stale data,
> but personally I would always recommend to run with
> torque.objectIsCaching = false anyway.
Well, if one really cares about transaction safety, getting stale data
_IS_ a problem. A big one. Certainly not if one wants to generate dynamic
content for a web site, but remember the bank account scenarios in the sql
textbooks ?
Hm, personally, I always use torque.objectIsCaching = true. The reason is
as follows: In most applications, I need transaction safety. Then, I like
to separate the application into a data accesss layer and the presentation
layer. But the presentation layer should be able to retrieve associated
objects, in the bookstore example this would be e.g. to be able to ask a
book for its author. If I want to make this transaction safe and would
have set torque.objectIsCaching to false, then I would have to pass the
database connection to the presentation layer, which is not what a
presentation layer is about.
>
>> 3) add a generator property torque.silentDbFetch, which is true by default.
>> If set to false, the getXXX() methods without arguments will not access the
>> db silently, but return null if the associated objects has not been read
>> yet. Also, the method does not throw a Torque exception any more (which is
>> the main reason why using these methods is more convenient than
>> getXXX(boolean silentDbFetch) with silentDbFetch set to false).
>
> That sounds like the best way to go.
>
Yes, I also think this one is crucial. But the same as for the overcrowded
classes goes for the generator properties: If there are too many of them,
we have to make sure that every combination of them works as expected. The
runtimetest does not ensure this automatically, which is a problem. One
day I should look into being able to run the runtimetest for different
combinations of generator properties sequentially.
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: silent db hit for associated objects
Posted by "Henning P. Schmiedehausen" <hp...@intermeta.de>.
Thomas Fischer <fi...@seitenbau.net> writes:
>Hi,
>If one has an association between objects by foreign key, (e.g. book ->
>author association in the tutorial), Torque generates methods to get the
>associated objects (e.g. Book.getAuthor() and Author.getBooks()). If the
>associated object has not yet been retrieved, these methods silently ask
>the database for the objects, using a new connection from the database
>pool.
>This is not at all what one wants if one wants to use transactions in an
>application, because one wants to do all reads in the same transaction. I
>therefore plan to do the following
>1) add a method getXXX(boolean silentDbFetch), which only hits the database
>silently if silentDbFetch is set to true. If set to false and the
>associated object(s) have not been retrieved yet, null is returned, even
>though there might be associated objects in the database.
-0 The classes are already too overloaded IMHO.
>2 ) add a method getXXX(connection) for the n->1 direction, whcih uses the
>provided connection if a db hit is necessary (in the book example, this
>would be book.getAuthor(connection) (author.getBooks() was already present)
+0; this is better, but still not really good. The question is: Does it
hurt? The database hit is read-only, so a rollback would only be
necessary if you have some trigger in the database which does writing
changes on a read-access. In the worst case you will get stale data,
but personally I would always recommend to run with
torque.objectIsCaching = false anyway.
>3) add a generator property torque.silentDbFetch, which is true by default.
>If set to false, the getXXX() methods without arguments will not access the
>db silently, but return null if the associated objects has not been read
>yet. Also, the method does not throw a Torque exception any more (which is
>the main reason why using these methods is more convenient than
>getXXX(boolean silentDbFetch) with silentDbFetch set to false).
That sounds like the best way to go.
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/
RedHat Certified Engineer -- Jakarta Turbine Development -- hero for hire
Linux, Java, perl, Solaris -- Consulting, Training, Development
4 - 8 - 15 - 16 - 23 - 42
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org