Wednesday, 14 November 2007

Finding per Thread cpu utilisation of a java process on linux

The problem I often face is trying to find out which thread in my App Server is using all my cpu. The way to do this is to get the per thread cpu utilisation using ps and then find out what each thread is going by sending the java process a kill -SIGQUIT signal. - obtain the per thread cpu utilisation on linux using: ps -eLo pid,lwp,pcpu,cmd This will give the following columns headers: PID (Process Id), LWP (lightweight process id), %CPU (cpu utilisation), CMD (process command) - send the java process you are interested in a SIGQUIT signal: kill -SIGQUIT This will print out a full thread dump into the stdout of your java process: Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_09-b01 mixed mode): "UIL2(SocketManager.MsgPool@45a36521 client=154.1.40.146:8093)#17" daemon prio=1 tid=0x0000002b68d5baf0 nid=0x2bd7 in Object.wait() [0x000000005a9d3000..0x000000005a9d3b30] at java.lang.Object.wait(Native Method) - waiting on <0x0000002acc2f0bf0> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:353) - locked <0x0000002acc2f0bf0> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747) at java.lang.Thread.run(Thread.java:595) "UIL2(SocketManager.MsgPool@4ce52059 client=154.1.40.146:8093)#18" daemon prio=1 tid=0x0000002b70d0a880 nid=0x2bd6 in Object.wait() [0x0000000057ba5000..0x0000000057ba5bb0] at java.lang.Object.wait(Native Method) - waiting on <0x0000002acc2f0bb0> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:353) - locked <0x0000002acc2f0bb0> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747) at java.lang.Thread.run(Thread.java:595) - from the output of the ps command locate the thread with the highest cpu utilisation for you given process. convert the lwp id into a hex value. Find the thread dump of the thread with nid (native thread id) equal to the hex value of the lwp  id in the ps command.