The HotSpot Serviceability Agent (SA) is a set of Java APIs which mirror the internal APIs of the HotSpot VM and which can be used to examine the state of a HotSpot VM.
The system understands the layout of certain VM data structures and is able to traverse these structures in an examination-only fashion; that is, it does not rely on being able to run code in the target VM. For this reason it transparently works with either a running VM or a core file.
The system can reconstruct information about Java frames on the stack and objects in the heap. Many of the important data structures in the VM like the CodeCache, Universe, StubQueue, Frames, and VFrames have been exposed and have relatively complete (or at least useful) implementations.
A small graphical debugger called HSDB (the "HotSpot Debugger") has been written using these APIs. It provides stack memory dumps annotated with method invocations, compiled-code inlining (if present), interpreted vs. compiled code, interpreter codelets (if interpreted), and live oops from oop-map information. It also provides a tree-based oop inspector. More information will be added as necessary; please send email with suggestions on what would be useful.
The SA currently only works on Solaris. It uses dbx to connect to the remote process or core file and communicates with a small piece of code (an "import module") loaded into the debugger.
The Java-side source code, which is the bulk of the SA, is in src/share/vm/agent. The organization of the sun.jvm.hotspot package hierarchy mirrors the organization in the VM. This should allow engineers familiar with the HotSpot sources to quickly understand how the SA works and to make modifications if necessary. To build these sources, cd to src/share/vm/agent and type "make".
The SA on Solaris works by communicating with a small piece of code (an "import module") loaded into dbx. The source code for this import module is in src/os/solaris/agent. To build this library, cd to src/os/solaris/agent and type "make 32bit" or "make 64bit". The distinction is necessary because the SPARC version of dbx ships with two versions of its executable, and depending on which architecture (v8 or v9) the debugger is running on selects the appropriate executable. The SA tries the v8 version first, but if you are running on a v9 machine you must provide both versions to the SA.
The system is currently hardwired to look on jano for its dbx executable and import module. The relevant directory structure looks like this:
The code which builds up path names to these executables is contained in sun.jvm.hotspot.HotSpotAgent.java. There are hardcoded paths in this file to jano, but the rest of the system is isolated from this.
(It would be nice to have a panel in the debugger allowing configuration of all of the known operating systems' options; right now Solaris is the only supported OS, making that easier.)
An installation of HSDB has been placed on jano. To access it, add the following directory to your PATH:
/net/jano/export/disk05/hotspot/sa/bin/common
To start the debugger, type "hsdb".
Alternatively, you can start a local build of the debugger by building the sources in src/share/vm/agent, cd'ing to that directory, and typing "java sun.jvm.hotspot.HSDB".
There are three modes for the debugger: attaching to a local process, opening a core file, and attaching to a remote "debug server". The remote case requires two programs to be running on the remote machine: the rmiregistry (see the script "start-rmiregistry" in this directory; run this in the background) and the debug server (see the script "start-debug-server"), in that order. start-rmiregistry takes no arguments; start-debug-server takes as argument the process ID or the executable and core file names to allow remote debugging of. Make sure you do NOT have a CLASSPATH environment variable set when you run these scripts. (The classes put into the rmiregistry are in sun.*, and there are permissions problems if they aren't placed on the boot classpath.)
NOTE that the SA currently only works against VMs on Solaris/SPARC. Remote debugging of Solaris/SPARC VMs on arbitrary platforms is possible using the debug server; select "Connect to debug server..." in HSDB.
Once the debugger has been launched, the threads list is displayed. The current set of functionality allows:
More functionality is planned. Please send email with suggestions on what would be useful, with any questions or comments, or if the debugger crashes.
HSDB does not suspend the system at a safepoint, but at an arbitrary point. This means that many of the invariants in the VM's code are not followed.
As it happens, most of the places where the code ported over from the VM has failed involve the topmost frame on the stack. Some modifications have been made to allow the system to recognize problematic situations.
Certainly, not all of the failure modes of the debugger have been found. Please send email if HSDB throws an exception. The best debugging aid in these situations is a core file since it is a static view of the VM to which we can then adapt the debugger code, as opposed to having to try to suspend the VM over and over to reproduce the failure. gcore (1) is a useful tool. (NOTE: do not try gcore with any application using the DGA X server extension (example: Java2Demo); the kernel will panic. See bug 4343237.)