Fetch notifications

[Matthew Adams <ma...@matthewadams.me>]

Hi all,

Sorry for the delay on this.  This is pursuant to my action item:
> [Feb 13 2007] AI Matthew discuss on email method by which an application
> might dynamically tune itself by tracking field access by use-case?

Overview

Currently, the only standard means to tune JDO applications is via
source code modifications, particularly via fetch plan functionality
offered by PM and vendor-specific APIs.  IMHO, performance tuning
could and should be addressed outside of the source code; it is a
concern orthogonal to the application's functionality, much like
persistence itself is to the domain objects.  The impetus behind these
proposals, then, is to allow for performance tuning capabilities to
take effect outside of code, or to provide JDO users or tooling
information on performance-related aspects, so that after performance
analysis, the data collected could be easily used to define fetch
groups that are use case-specific.  Since the only JDO standardized
performance tuning feature that I could see is the fetch plan, I will
start there (solution A below) and progress to a more general solution
(B & C below).  These are only food for thought; I'm hoping that the
input from others will help standardize performance tuning features
commonly available among implementations.

Solution A:  FetchListener

My first thought, which doesn't consider the notion of "use case", was
to define an interface an implementation of which could register
itself with the PM/PMF that would be notified when a given field of a
given object was loaded.

For example,

public interface FetchListener {
  void onFetch(FetchEvent event);
}

public class FetchEvent {
  Object getInstance() {...} // returns the PC whose field was fetched
  Class getInstanceClass() { return getInstance().getClass(); }
  String getInstanceClassName() { return getInstanceClass().getName(); }
  Field getField() {...} // returns the Field object for the field
being retrieved
  String getFieldName() {...} // returns the name of the field being retrieved
  Object getFieldValue() {...} // returns the value of the field being retrieved
  String[] getFetchGroups() {...} // returns the name(s) of the fetch
group(s) currently in use, or an empty array if none
}

// mods to PersistenceManager
public interface PersistenceManager {
  ...
  void addFetchListener(FetchListener fl);
  void removeFetchListener(FetchListener fl);
  ...
}

The spec itself would not necessarily provide a default implementation
of FetchListener, but I could see the Apache JDO project providing a
convenient, configurable FetchListener implementation that could be
interrogated for the fields accessed during the PM's use so that a
fetch group could be easily obtained, named, and added to the JDO
metadata by the developer.

Solution B:  StateManagerListener
Next, I considered instead a more general solution that would use the
StateManager interface and the idea of a StateManagerListener, with
notifications for each applicable method on the StateManager.

For example,

public interface StateManagerListener {

  void onGetBooleanField(StateManagerEvent event);
  void onGetByteField(StateManagerEvent event);
  ...
  void onProvidedBooleanField(StateManagerEvent event);
  void onProvidedByteField(StateManagerEvent event);
  ...
  void onReplacingBooleanField(StateManagerEvent event);
  void onReplacingByteField(StateManagerEvent event);
  ...
  void onSetBooleanField(StateManagerEvent event);
  void onSetByteField(StateManagerEvent event);
  ...
}

public class StateManagerEvent event {
  ... // TBD, but probably similar to FetchEvent
}

Solution C:  JDO Use Case
Introduce a new concept to JDO called "use case" that encompasses
exclusively performance-tuning related features, including an
extension mechanism so that tuning features that aren't specified can
still be supported.  This design is similar to Xcalia's use case
support.  It would look something like the following:

// mods to PersistenceManager
public interface PersistenceManager {
  ...
  void beginUseCase(String useCaseName); /* named performance tuning
settings in metadata (fetch plan, locking, optimistic txns, etc) would
be set and used */
  void endUseCase(); /* unsets the current use case's settings; no-op
if no current use case */
  ...

UseCase could also be a standard PMF property of type String that
would hold for all PMs obtained from the PMF, which would allow use
case-specific PMFs and no need for a user to call the
beginUseCase/endUseCase methods programmatically.

Let's kick it around!

-matthew

Re: Fetch notifications

[Matthew Adams <ma...@matthewadams.me>]

inline...

On Sat, Mar 28, 2009 at 4:33 AM, Andy Jefferson <an...@datanucleus.org> wrote:
> By "tuning", are you proposing addition of non-fetchplan fields to the
> selected fetch plan, or are you also proposing removal of "unused" fields
> from the selected fetch plan?
>
I had only explicitly considered the addition of fields that hadn't
been loaded, but it could be used to prevent over-loading (loading
more than is necessary for the given use case).  Me likey.

> If the proposal includes the removal of unused fields from the fetch plan then
> the listener would need to be on any jdoGetXXX/jdoPutXXX calls (rather than
> when something was loaded). So would be an AccessListener - do we need to
> distinguish between user read and user write of a field for this? Dont think
> so.
>
Yes, it would need to be on all jdoGet/jdoPut calls.  As I mentioned
in my previous email, I think a nice, general StateManagerListener
would be best.  Let a listener listen to anything it wants to -- no
need to limit them.

> Presumably somewhere the fetch plan is being updated as a result of the fetch
> notifications, so a new fetch group is added "AUTO" with the fetched fields
> in ?
>
That would be nice, but I think that begins to step beyond what the
spec should mandate.  The spec could suggest this, but I think that
should be as far as it goes.

> Who does the "tuning" ? The implementation ? Something generic in
> JDO(Impl)Helper?
>
I'm thinking that we could write an implementation that is not part of
the specification deliverables but that is part of the Apache JDO
project.  The initial idea would be to register it as a listener and
then, after the use case has been exercised (during integration or
performance testing), the listener could be interrogated for what it
would recommend for fetch plans.

>
> Re: UseCase
> +1 to the general idea. Needs definition of what is in scope for inclusion in
> the "use-case". Obviously current PM props
> ("multithreaded", "detachAllOnCommit", "IgnoreCache", "queryTimeoutInMillis", "copyOnAttach", "fetchPlan")
> and current Transaction props (optimistic, nontx-read, nontx-write,
> restoreValues, retainValues, txnIsolationLevel, txnRollbackOnly) and also
> extensions.
>
Yes, we would have to define what's in scope.  I agree on most of
these, but I was a little surprised by multithreaded -- I suppose it's
performance related, but I always thought of it more as a functional
thing.  Maybe it's the gray area between the two.  Specified
transaction properties are definitely candidates for inclusion in use
case definitions.  One thing that I would expect to be an extension in
almost every implementation would be locking semantics.  JDO doesn't
really deal much with locking directly, but the implementations do.
Maybe JDO could define the 4 standard, loosely-defined semantics
(SERIALIZABLE, READ_COMMITTED, REPEATABLE_READ, READ_UNCOMMITTED) for
the purposes of use case definition.

> Is the use-case definable solely in XML ? or perhaps we have an API too.
>
I would expect the primary manifestation to be in metadata outside the
code (XML or otherwise).  I suppose we could have an API for
completeness, but I'm curious as to how API-based use cases would be
utilized.

> Is there only one use-case active and beginUseCase() removes any previous ?
>
Correct.  One use case active at a time, and yes, a begin without a
prior end would implicitly end the previously active use case.  An
explicit end call would end whichever use case was currently active,
and be a no-op if no use case were currently active.

> I'd probably prefer
> PM.setUseCase(String name);
> PM.unsetUseCase();
>
Pot-AY-toes, pot-AH-toes.  To have a bean-style property "useCase" on
PM (& PMF) seems intuitive.  I might require that if useCase is set to
null or the empty string, it effectively stops using the current use
case settings, and I would implement unsetUseCase as a convenience
method:  public void unsetUseCase() { setUseCase(null); }.

-matthew

Re: Fetch notifications

[Andy Jefferson <an...@datanucleus.org>]

Hi,

By "tuning", are you proposing addition of non-fetchplan fields to the 
selected fetch plan, or are you also proposing removal of "unused" fields 
from the selected fetch plan?

If the proposal includes the removal of unused fields from the fetch plan then 
the listener would need to be on any jdoGetXXX/jdoPutXXX calls (rather than 
when something was loaded). So would be an AccessListener - do we need to 
distinguish between user read and user write of a field for this? Dont think 
so.

Presumably somewhere the fetch plan is being updated as a result of the fetch 
notifications, so a new fetch group is added "AUTO" with the fetched fields 
in ?

Who does the "tuning" ? The implementation ? Something generic in 
JDO(Impl)Helper?


Re: UseCase
+1 to the general idea. Needs definition of what is in scope for inclusion in 
the "use-case". Obviously current PM props 
("multithreaded", "detachAllOnCommit", "IgnoreCache", "queryTimeoutInMillis", "copyOnAttach", "fetchPlan")
and current Transaction props (optimistic, nontx-read, nontx-write, 
restoreValues, retainValues, txnIsolationLevel, txnRollbackOnly) and also 
extensions.

Is the use-case definable solely in XML ? or perhaps we have an API too.

Is there only one use-case active and beginUseCase() removes any previous ?

I'd probably prefer
PM.setUseCase(String name);
PM.unsetUseCase();



-- 
Andy  (DataNucleus - http://www.datanucleus.org)