Comments on: My memory leak in pictures http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures Thu, 01 Jun 2017 18:51:00 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Using Transfer’s new EHCache » ghidinelli.com http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-58921 Using Transfer’s new EHCache » ghidinelli.com Thu, 19 Nov 2009 05:29:49 +0000 http://www.ghidinelli.com/?p=907#comment-58921 [...] Delete all of your *.transfer files from your generated folder and restart your application. You should see it come back up, just like it always has. The difference? No more leaky SoftReferences! [...] [...] Delete all of your *.transfer files from your generated folder and restart your application. You should see it come back up, just like it always has. The difference? No more leaky SoftReferences! [...]

]]> By: Brian http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-58827 Brian Thu, 12 Nov 2009 00:51:22 +0000 http://www.ghidinelli.com/?p=907#comment-58827 @Chris - I did work out the root cause with Mark Mandel unfortunately there was no resolution. It ultimately wound up in this: http://www.ghidinelli.com/2009/09/21/match-bounty-transfer-cache-improvements which succeeded and the work is currently underway. We should have something to test here in the next few days and then goodbye internal Transfer cache and hello memory stability! FWIW, moving back from JDK 1.6.0_u14 to u10 made the app last about 3x longer (roughly a week at a go instead of ~2 days). No reason why that should be the case but... that's where we are today, waiting for Mark to finish the caching update. @Chris – I did work out the root cause with Mark Mandel unfortunately there was no resolution. It ultimately wound up in this: http://www.ghidinelli.com/2009/09/21/match-bounty-transfer-cache-improvements which succeeded and the work is currently underway. We should have something to test here in the next few days and then goodbye internal Transfer cache and hello memory stability!

FWIW, moving back from JDK 1.6.0_u14 to u10 made the app last about 3x longer (roughly a week at a go instead of ~2 days). No reason why that should be the case but… that’s where we are today, waiting for Mark to finish the caching update.

]]> By: Chris Peterson http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-58818 Chris Peterson Wed, 11 Nov 2009 19:13:44 +0000 http://www.ghidinelli.com/?p=907#comment-58818 Brian, Did you ever work out what in Transfer was keeping that reference, and get a solid resolution to this issue? Just curious what you ended up with! Chris Peterson Brian,

Did you ever work out what in Transfer was keeping that reference, and get a solid resolution to this issue? Just curious what you ended up with!

Chris Peterson

]]> By: ubuntubugs: [440985] ld thrashes system while compiling following … | DevBlogr http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-58331 ubuntubugs: [440985] ld thrashes system while compiling following … | DevBlogr Sun, 04 Oct 2009 22:15:31 +0000 http://www.ghidinelli.com/?p=907#comment-58331 [...] My memory leak in pictures » ghidinelli.com [...] [...] My memory leak in pictures » ghidinelli.com [...]

]]> By: brian http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57964 brian Wed, 09 Sep 2009 17:25:36 +0000 http://www.ghidinelli.com/?p=907#comment-57964 It's worth noting that the bug 72641 mentioned above was actually fixed in CHF2, which we had applied previously, and which appeared to do nothing for our particular issue. Tentatively, we have radically improved our situation by going from 1.6.0 U12 and U14 back to U10. With that change we have been up for over a week without a restart and still have 30% free memory. I don't think it has solved the issue but we're making progress and at least I don't have to babysit the servers ever 48 hours. Mark Mandel can't reproduce the issue despite having seen it in my heap dump so we're struggling with how to actually fix. It’s worth noting that the bug 72641 mentioned above was actually fixed in CHF2, which we had applied previously, and which appeared to do nothing for our particular issue.

Tentatively, we have radically improved our situation by going from 1.6.0 U12 and U14 back to U10. With that change we have been up for over a week without a restart and still have 30% free memory. I don’t think it has solved the issue but we’re making progress and at least I don’t have to babysit the servers ever 48 hours. Mark Mandel can’t reproduce the issue despite having seen it in my heap dump so we’re struggling with how to actually fix.

]]> By: Brian http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57842 Brian Tue, 01 Sep 2009 18:10:10 +0000 http://www.ghidinelli.com/?p=907#comment-57842 Wow! This is a big update, thanks for posting here Brett! I'm going to apply it tonight. An uUpdate on working with Mark - turns out he had a faulty unit test so we don't have a culprit yet. We know what's holding onto the references (reapMap) but not why. Wow! This is a big update, thanks for posting here Brett! I’m going to apply it tonight. An uUpdate on working with Mark – turns out he had a faulty unit test so we don’t have a culprit yet. We know what’s holding onto the references (reapMap) but not why.

]]> By: Brett http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57841 Brett Tue, 01 Sep 2009 13:48:45 +0000 http://www.ghidinelli.com/?p=907#comment-57841 @Steve - I never tested it with Application.cfc but that would be a huge hotspot if it had the same issue. I just tested the new Cumulative Hot Fix 3 for ColdFusion 8.0.1 and this Issue is finally fixed. I assume this is the fix mentioned for bug 72641 "Fix for memory leaks with CFCs stored in memory scopes. Note, this does not eliminate need for proper use of VAR scope in CFC methods." http://kb2.adobe.com/cps/511/cpsid_51180.html @Steve – I never tested it with Application.cfc but that would be a huge hotspot if it had the same issue.

I just tested the new Cumulative Hot Fix 3 for ColdFusion 8.0.1 and this Issue is finally fixed. I assume this is the fix mentioned for

bug 72641
“Fix for memory leaks with CFCs stored in memory scopes. Note, this does not eliminate need for proper use of VAR scope in CFC methods.”

http://kb2.adobe.com/cps/511/cpsid_51180.html

]]> By: Steve 'Cutter' Blades http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57766 Steve 'Cutter' Blades Thu, 27 Aug 2009 22:13:36 +0000 http://www.ghidinelli.com/?p=907#comment-57766 @Brian - Glad you've figured it out. I look forward to seeing how you finally tracked it down, and what it took to fix it. @Brett - Interesting link there. I'm especially interested in this statement: "Components should not be set assigned to sessions from within another component" I would think that would include Application.cfc, which is where we place many different components into the SESSION scope, within the onSessionStart() method. (Glad to hear it's resolved in CF9, but I imagine some folks have a more immediate need...) @Brian -

Glad you’ve figured it out. I look forward to seeing how you finally tracked it down, and what it took to fix it.

@Brett -

Interesting link there. I’m especially interested in this statement:

“Components should not be set assigned to sessions from within another component”

I would think that would include Application.cfc, which is where we place many different components into the SESSION scope, within the onSessionStart() method. (Glad to hear it’s resolved in CF9, but I imagine some folks have a more immediate need…)

]]> By: Brett http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57765 Brett Thu, 27 Aug 2009 20:49:19 +0000 http://www.ghidinelli.com/?p=907#comment-57765 These are a lot of "fun", glad you guys have uncovered the culprit. I'm curious to hear what was causing it. We ran into a memory leak in CF8 about a year ago, it took a while to trace it down but I isolated it in about 50 lines of code. It does not look like the problem above, our issue was that there were components leaking into sessions which on a high traffic site ate all our memory. The bad news is the issue still exists in CF8, the good news is it appears fixed in CF9. If anyone is interested, I posted the code to reproduce it here: http://drop.io/memleak You'll want JMC memleak running when you run the code so you can see the number of instances of a locally scoped CFC persist for the life of the session. Hope it helps someone - it was a week I don't want to relive. These are a lot of “fun”, glad you guys have uncovered the culprit. I’m curious to hear what was causing it.

We ran into a memory leak in CF8 about a year ago, it took a while to trace it down but I isolated it in about 50 lines of code.

It does not look like the problem above, our issue was that there were components leaking into sessions which on a high traffic site ate all our memory.

The bad news is the issue still exists in CF8, the good news is it appears fixed in CF9.

If anyone is interested, I posted the code to reproduce it here:

http://drop.io/memleak

You’ll want JMC memleak running when you run the code so you can see the number of instances of a locally scoped CFC persist for the life of the session.

Hope it helps someone – it was a week I don’t want to relive.

]]> By: marc esher http://www.ghidinelli.com/2009/08/25/my-memory-leak-in-pictures/comment-page-1#comment-57740 marc esher Wed, 26 Aug 2009 11:36:14 +0000 http://www.ghidinelli.com/?p=907#comment-57740 Brian, when I mentioned increasing frequency, I meant adding something like this to your jvm settings: -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 which would cause GC every 10 minutes. That's great news that you guys have figured it out. I'd love to see how you used MAT to get to the answer, because like you, I find "getting to the object" to be somewhat dark art, somewhat hunched-over-till-your-neck hurts persistence. If there's an easier way, I'd love to learn about it. Brian,
when I mentioned increasing frequency, I meant adding something like this to your jvm settings: -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000

which would cause GC every 10 minutes.

That’s great news that you guys have figured it out. I’d love to see how you used MAT to get to the answer, because like you, I find “getting to the object” to be somewhat dark art, somewhat hunched-over-till-your-neck hurts persistence. If there’s an easier way, I’d love to learn about it.

]]>