"Snapshot" support

[Steve Poole <sp...@googlemail.com>]

One of the capabilities that this API is intended to provide is support for
"Snapshots"

This is  based on the idea that for various reasons the dumps that we can
get today can be too big, take too long to generate , not have the right
information etc.

Also we need to recognise that dumps are not only produced to help diagnose
a failure.  Some users consume dumps as part of monitoring a live system.

So we need to discuss (at least)

a)  How dump content configuration would work
b)  What sorts of data are needed in a snapshot dump

This is the largest outstanding piece of the API.   Now with Alois and
Andreas on board we can start to clarify usecases and  resolve the design


Cheers

Steve

Re: "Snapshot" support

[Steve Poole <sp...@googlemail.com>]

On Tue, Sep 8, 2009 at 3:53 PM, Stuart Monteith <st...@stoo.me.uk> wrote:

> I think the functionality to be addressed by this API and the JVM are:
>
> 1. When to generate the dump.
>

Agree


> 2. What to call the dump file.
>

Disagree

3. What should be in the dump.
>

Agree


> 4. What should do the dumping.
>
> half agree


> 1. When to generate the dump.
> There would be a few use cases met by just having a method that generates a
> dump file when it is called.
> In addition there may be certain events where we'd want a dump file
> generated.
> For example:
>   when exceptions are thrown.
>   when OOM occurs.
>   when a time interval has expired.
>
> Agree


> These kinds of features already exist. For example, there is already a JMX
> MBean that can be invoked to generate a heapdump.
> The IBM J9 JVM has features whereby dumps can be generated given certain
> command line parameters - these are non-standard and
> can't be set at runtime.
> Should we be providing this within the JMX infrastructure?
>
> I don't understand what would mean to the API.    Care to spell out your
vision ?


> 2. What to call the dump file.
> I think it might be useful if we can provide a template to name the file,
> much like core file configuration might be done on linux, etc.
>
> The trouble is that this is not about a "dump file"  its about calling a
standard mechanism to drive an implementation to do its thing -  there may
not be any files involved - or they may be many.
We could definitely consider providing some hint , and may be even getting
back a handle that could be used immediately by the API to access the
dump...



> 3. What should be in the dump.
> I think this will require some use cases. But to my rough thoughts are:
>
> * Enable callers to choose what types of information - e.g. threads?
> Monitors? objects+classes? memory? imagethreads?
> * Enable callers to identify certain entities in the JVM to dump, and dump
> only them.
>  - Dump thread "This name", or the thread "java.lang.Thread thisthread;"
>  - Dump classes in classloader "java.lang.Classloader loader;"
>  - Dump objects of certain types identified with java.lang.Class.
>  - Dump all objects from this point in the object graph up to n references
> away - a truncated object graph.
>  - Dump all objects "tagged" in a certain way using bytecode injection.
>
> Those are my rough ideas, how some of this would be specified are just
> details ^_^
>
> I have two questions  here

a) How does this definition get stored ?
b) Who executes the selection process?

The reason this isn't simply  answered as a)  by Kato  b) the Dump
Implementation    is that you've listed selection criteria that is likely
going to need JVM involement.    I supose it would be feasible to restrict
the selection process down to something that could by done by a JVMTI agent
-  reading your list again that may actually be the case.

What we really need is to look at some solid examples of what data is
required to solve a problem and see how you would need to define a matching
selection.





> 4. What should do the dumping.
>
> There may be multiple implementations of the snapshot API available.
> I'd hope that for each JVM there'd be a fully-featured, highly integrated
> implementation. However, that might not be the case.
> One way to select an implementation would be by automatically choosing the
> implementation that provides the functionality that are being requested.
> e.g. an dump needing a memory dump would choose the implementation that uses
> core dumps.
> An alternative would be to choose them by name, or by class.
>
> Sort of agree but I suspect the devil is in the details.    I think we
really need one or two examples to work through.    Lets look at what we've
got , pick one and explore...


> Regards,
>
>   Stuart
>
>
>
>
>
>
> Steve Poole wrote:
>
>> One of the capabilities that this API is intended to provide is support
>> for
>> "Snapshots"
>>
>> This is  based on the idea that for various reasons the dumps that we can
>> get today can be too big, take too long to generate , not have the right
>> information etc.
>>
>> Also we need to recognise that dumps are not only produced to help
>> diagnose
>> a failure.  Some users consume dumps as part of monitoring a live system.
>>
>> So we need to discuss (at least)
>>
>> a)  How dump content configuration would work
>> b)  What sorts of data are needed in a snapshot dump
>>
>> This is the largest outstanding piece of the API.   Now with Alois and
>> Andreas on board we can start to clarify usecases and  resolve the design
>>
>>
>> Cheers
>>
>> Steve
>>
>>
>>
>
> --
> Stuart Monteith
> http://blog.stoo.me.uk/
>
>

Re: "Snapshot" support

[Stuart Monteith <st...@stoo.me.uk>]

I think the functionality to be addressed by this API and the JVM are:

1. When to generate the dump.
2. What to call the dump file.
3. What should be in the dump.
4. What should do the dumping.

1. When to generate the dump.
There would be a few use cases met by just having a method that 
generates a dump file when it is called.
In addition there may be certain events where we'd want a dump file 
generated.
For example:
    when exceptions are thrown.
    when OOM occurs.
    when a time interval has expired.

These kinds of features already exist. For example, there is already a 
JMX MBean that can be invoked to generate a heapdump.
The IBM J9 JVM has features whereby dumps can be generated given certain 
command line parameters - these are non-standard and
can't be set at runtime.
Should we be providing this within the JMX infrastructure?

2. What to call the dump file.
I think it might be useful if we can provide a template to name the 
file, much like core file configuration might be done on linux, etc.

3. What should be in the dump.
I think this will require some use cases. But to my rough thoughts are:

 * Enable callers to choose what types of information - e.g. threads? 
Monitors? objects+classes? memory? imagethreads?
 * Enable callers to identify certain entities in the JVM to dump, and 
dump only them.
   - Dump thread "This name", or the thread "java.lang.Thread thisthread;"
   - Dump classes in classloader "java.lang.Classloader loader;"
   - Dump objects of certain types identified with java.lang.Class.
   - Dump all objects from this point in the object graph up to n 
references away - a truncated object graph.
   - Dump all objects "tagged" in a certain way using bytecode injection.

Those are my rough ideas, how some of this would be specified are just 
details ^_^

4. What should do the dumping.

There may be multiple implementations of the snapshot API available.
I'd hope that for each JVM there'd be a fully-featured, highly 
integrated implementation. However, that might not be the case.
One way to select an implementation would be by automatically choosing 
the implementation that provides the functionality that are being 
requested. e.g. an dump needing a memory dump would choose the 
implementation that uses core dumps.
An alternative would be to choose them by name, or by class.

Regards,

    Stuart





Steve Poole wrote:
> One of the capabilities that this API is intended to provide is support for
> "Snapshots"
>
> This is  based on the idea that for various reasons the dumps that we can
> get today can be too big, take too long to generate , not have the right
> information etc.
>
> Also we need to recognise that dumps are not only produced to help diagnose
> a failure.  Some users consume dumps as part of monitoring a live system.
>
> So we need to discuss (at least)
>
> a)  How dump content configuration would work
> b)  What sorts of data are needed in a snapshot dump
>
> This is the largest outstanding piece of the API.   Now with Alois and
> Andreas on board we can start to clarify usecases and  resolve the design
>
>
> Cheers
>
> Steve
>
>   

-- 
Stuart Monteith
http://blog.stoo.me.uk/