JVM Core Dumping

The work I do for Fivium is mostly developing a Java servlet called FOXopen which is a stateful web framework that typically runs on top of Tomcat.

Unfortunately, as with all software, it occasionally crashes and hangs (typically using 100% CPU stuck in a GC loop). When a Java application has hung because of underlying memory issues the best way to investigate the cause would be to dump all the memory, a Heap Dump.

Even more unfortunate perhaps is that when the JVM is hung you often cannot get a heap dump using the conventional tools or methods. And tools which might work may take extraordinary amounts of time to complete.

Recently I stumbled upon a method of getting a heap dump which is surprisingly fast so I figured I’d share it here.

Typically when a JVM is running you can use a tool called jmap, which comes with every JDK install in the bin folder, to attach to the running JVM and either find out heap statistics or generate a full heap dump to analyse.

While reading the jmap documentation however I noticed it can take a core file instead of the PID of the Java process to map. Researching online it seems that if you run on Linux you can generate a core dump from the running process and have jmap analyse that.

More interesting for me was finding out that you can get a core dump from a process even when it’s completely hung.

Here’s how to do this:

1. Find the PID of the hung process

• If you use top you can see the PID in the leftmost column typically
• Otherwise I tend to run ps aux | grep java replacing Java occasionally with an argument I know was passed (e.g. part of the Tomcat path or the word catalina which Tomcat uses a lot)

2. Attach The GNU Project Debugger(GDB) to the process

• run sudo gdb --pid=12345 where 12345 is the PID of the process you want to get a core dump for
• You should now have a(gdb) prompt
• Run gcore /opt/jvm.core to dump the memory into a file at /opt/jvm.core (Make sure you have enough disk space, the core dump will be the full size of the memory the JVM was using at the time you run this)
• Run detach to detach the debugger
• Run quit to leave the debugger and get back to your regular Linux prompt

3. You now have the entire memory of the JVM process saved in that core file, so if you need to you can restart Tomcat or whatever you might normally do to your Java process to fix it, we can investigate any time we like now

To investigate the core file we first need to convert it. A core dump contains all the Java Heap, with various memory pools, but it also contains the internal memory and state of the JVM. Java Heap analysis tools typically can’t work with a raw core dump and need a Heap Profile instead. This is where jmap comes in, taking the core as an argument.

As a quick test to check the core dump was correctly made and from a valid Java I tend to run sudo jmap -clstats /usr/bin/java /opt/jvm.core first (this should just dump classloader stats to stdout). If that runs without error it’s time to get a Java Heap dump out of the core.

Run sudo jmap -dump:format=b,file=jvm.hprof /usr/bin/java /opt/jvm.core

This may take a while, but it should eventually say “Heap dump file created” and you should have a jvm.hprof file.

You can now use tools like YourKit or VisualVM (which also comes as standard with the JDK in the bin folder) or jhat (command line tool also in the JDK bin folder) to open up that hprof file and inspect all the instances of your objects. Hopefully you can track down your memory leak or find out what memory values were causing your issues.

One issue I have run across with this technique is that jmap MUST run with the JVM which the core file came from as an argument, /usr/bin/java in my example commands. If you use Tomcat with JSVC then this is, as far as I currently know, impossible to do. If your machine just has multiple Java installs however, you just need to make sure you pass in the right one when running jmap.

I also read there are issues using gcore with IBM Java which you might need to be aware of.