Comments on: Finding memory leaks in your ColdFusion JVM

By: Brian

Brian — Fri, 22 Jan 2010 15:31:07 +0000

@Michael – glad it helped! It was A great learning experience and ultimately helped prove that softreferences under java in Tranfer ORM weren’t freeing up memory like they should. That led to a sponsored open source development project for Mark and now Transfer has a pluggable cache architecture which has solved our memory leak issues. It is frustrating that it took so long to diagnose and solve but still better than rewriting the site or buying lots of new servers. Good luck!

By: Michael Horne

Michael Horne — Fri, 22 Jan 2010 13:42:35 +0000

Well, Brian, what can I say? After using the methods above, I’ve learnt a lot about the JVMs and identified some memory leaks in our code. This was sufficient to allow the process to run that was giving me major headaches.

I might even use the JRockit JVM in production as it seems to be a bit zippier than the standard one.

I found I could monitor the JVM remotely using the Mission Control plugin for Eclipse (which I installed in CF Builder).

I did find when I installled JRockit Mission Control v3.1.2 for Java 5 on our 64-bit platform that I had to copy the amd64 folder in jre\lib to i386 in the same folder. Otherwise, no valid JVM can be found. This is probably just a bad combination of circumstance tho!

Thanks again for this brilliant post
–
Mike Horne

By: Michael Horne

Michael Horne — Wed, 20 Jan 2010 11:06:56 +0000

Wow.
Fantastic post. I’d seen the Jrock stuff before, but badly explained. Will give this a go – I have a 2gb 64-bit instance that keeps running out of memory!

By: charlie arehart

charlie arehart — Mon, 27 Jul 2009 20:57:34 +0000

Great to hear (and no worries about covering CF7 stuff: those last two paragraphs were addressed to “anyone else” who may want to pick up that baton. I’ll ping you in mid-august. Thanks.

By: Brian

Brian — Mon, 27 Jul 2009 19:57:21 +0000

@Charlie – sure thing; I’m really busy for the next couple of weeks but ping me mid-August and let’s set a date. I probably can’t cover the CF7 stuff unfortunately; I don’t really touch it any longer and don’t have the time to invest in figuring out what works and what doesn’t.

By: charlie arehart

charlie arehart — Mon, 27 Jul 2009 02:17:00 +0000

Great stuff here, Brian, of course. Thanks for organizing and sharing it.

Would you be at all interested in offering a informal presentation on the topic, where you basically just walk through this stuff live, on the Online CF Meetup? Since it’s recorded, I’m sure it would be very popular. You don’t need to “put together a presentation”. You could literally just walk through he process you describe here, live. If you’re at all interested, just let me know.

Same goes for anyone else reading this entry who may want to, either if Brian doesn’t, or if you have some other twist.

For instance, I’m sure some would love to hear about doing this in any form possible while still on CF 7, which of course is Java 1.4, which lacks so many tools, though it does have at least some. If anyone has shown folks how to to tackle memory leaks (other than the obvious file upload one fixed in in CF8 or available as a hotfix), I’m sure there would be an audience for that, too.

By: Steven Erat

Steven Erat — Fri, 17 Jul 2009 19:38:03 +0000

Thanks for the quick feedback Brian. I set the Trend preference to 0 but still don’t see cf objects with filter “cf*”. I do however see lots jrun objects. I’ll try running a larger app like blogcfc to see if that makes a difference.

I ran the Eclipse MAT on a CF9 heap dump. Wow, there’s an incredible abundance of information there. Love the table of contents view better than the Overview view. Will need to spend some time there just to understand how to best interpret the results.

By: brian

brian — Fri, 17 Jul 2009 19:23:51 +0000

@Steven – thanks for the headsup about the GC flag. I didn’t have that happen to me since my GC flags were different (although I too, ran u14 on CentOS 5.3).

On JRMC, there is an option somewhere that says to not limit small items in the heap; under preferences but I *thought* it was disabled by default. Go to Preferences -> Jrockit Mission Control -> Memory Leak Detector -> Trend and set the lowest heap usage to 0.

Also, a small side note, I don’t think regexps work so make sure it’s cf* not cf.*

By: Steven Erat

Steven Erat — Fri, 17 Jul 2009 19:07:31 +0000

Brian,

So I’ve got JRockit Mission Control running (jrmc) and it successfully attaches to ColdFusion (using CF9 but previously also did CF8). When I start MemLeak on the CF9 process id I see the view with all the Java objects. However, when I add a filter for “cf.*” the table clears and no data is shown. I noticed there’s a little warning at the bottom that says “ignoring objects that take less than 0.1% heap so I wonder if that could be why I don’t see cf objects. I’ve run some small demo apps and loaded the CF Admin, but nothing. Any ideas? Thanks!

By: Steven Erat

Steven Erat — Fri, 17 Jul 2009 18:23:26 +0000

Still going thru this and testing all the techniques. I learned that jstat and jmap are already available on Mac OS X, but they are the Apple implementations so for example jmap behaves differently such that it did not have the -dump switch

steven-erats-macbook-pro:~ stevenerat$ jmap -dump:format=b,file=/tmp/cf9.hprof 3432
Usage: jmap [option]
(to connect to a live java process)

On CentOS 5.3 Linux I installed the Sun 1.6.0_14 to get the utilities, then found that jmap didn’t work.

[steven@maccent ~]$ jmap -heap 31762
Attaching to process ID 31762, please wait…
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol “heapOopSize”

I remembered that the -XX:+UseParallelGC argument will not work with java debugging, so I changed it to -XX:+UseParNewGC and then it worked.