Memory fragmentation?
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 - 3 | Next > |
 |
Memory fragmentation?
by schrep-2
::
Rate this Message:
Hey Folks,
On yesterday's trunk build with around 18 tabs, including Zimbra, gmail, several google docs, the perf graphs, etc open I was at 264MB Working Set (. When I applied today's update and restarted with session restore, logged back into everything I'm at 140MB working set size. As far as I'm aware there are two possible causes for this difference in memory use: 1) Leaks 2) Heap fragmentation There has been a tremendous effort in finding leaks, the numbers posted recently about MochiTests running with few leaks, and we are running for quite some time through random URL's with few leaks. This leads me to wonder if the large difference here is a memory fragmentation problem here? Has anyone looked into this recently? Are there any easy ways to instrument the malloc to figure out how fragmented the heap is etc? Should I in general expect these two numbers to be very close to each other? Why does Schrep ask so many questions? Well strike that last one :-). _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by L. David Baron
::
Rate this Message:
On Thursday 2007-10-11 16:13 -0700, Mike Schroepfer wrote:
> On yesterday's trunk build with around 18 tabs, including Zimbra, gmail, > several google docs, the perf graphs, etc open I was at 264MB Working > Set (. When I applied today's update and restarted with session > restore, logged back into everything I'm at 140MB working set size. > > As far as I'm aware there are two possible causes for this difference in > memory use: For completeness, there are some other possibilities: > 1) Leaks I think this can be split into four possibilities: 1a) Cases where we allocate memory while viewing a page and never free it. These are the ones we've been focusing on. 1b) Cases where we allocate memory while viewing a page and don't free it until shutdown (or some other time significantly after leaving the page). 1c) Cases where we allocate memory while viewing a page and don't free it until the user leaves the page, but the page isn't holding on to something that requires us to keep the object allocated. 1d) Cases where the page is leaking memory -- i.e., accumulating arrays of javascript or DOM objects in global variables such that we can't free them (without solving the halting problem). > 2) Heap fragmentation There's also: 3) caches -- Things in various caches, particularly the back-forward cache, wouldn't necessarily be reloaded after restoring the session. Given that you were using gmail, Zimbra, google docs, etc., I'd be surprised if (1c) and/or (1d) weren't happening at least some here. It's probably worth looking at our memory use over time while using some of these apps. I think it's also worth looking at measuring fragmentation. -David -- L. David Baron http://dbaron.org/ Mozilla Corporation http://www.mozilla.com/ _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
|
|
|
|
|
|
|
|
|
Re: Memory fragmentation?
by L. David Baron
::
Rate this Message:
On Friday 2007-10-12 16:57 -0700, Mike Schroepfer wrote:
> > 1c) Cases where we allocate memory while viewing a page and don't > > free it until the user leaves the page, but the page isn't holding > > on to something that requires us to keep the object allocated. > > 1d) Cases where the page is leaking memory -- i.e., accumulating > > arrays of javascript or DOM objects in global variables such that we > > can't free them (without solving the halting problem). > > Closing the page would free memory in both cases then? Yes. Free in the sense of calling free(), which may or may not return the memory to the system (which could be due to fragmentation or due to just never returning the memory -- does anybody happen to know what the major malloc implementations do here?). > > Given that you were using gmail, Zimbra, google docs, etc., I'd be > > surprised if (1c) and/or (1d) weren't happening at least some here. > > It's probably worth looking at our memory use over time while using > > some of these apps. > > In theory if I close all tabs I should be at a similar state to a clean > start then? Exempting caches build up, leaks, and fragmentation that is. And, depending on what number you're measuring, free() not returning chunks of memory to the kernel even if they're entirely free. (I'm not sure how different malloc() implementations do in that regard.) That said, if a page has become completely unused it probably shouldn't be showing up in measures of resident set. (But I'm not sure if that actually happens on Windows unless you minimize the window. Or something.) -David -- L. David Baron http://dbaron.org/ Mozilla Corporation http://www.mozilla.com/ _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Jonas Sicking-2
::
Rate this Message:
Mike Schroepfer wrote:
> First off very helpful - comments inline. > >> >> 1b) Cases where we allocate memory while viewing a page and don't >> free it until shutdown (or some other time significantly after >> leaving the page). > > Right like bfcache and others. I think David was talking about a different type of 'leaks' here. We have had instances where objects end up getting stored in places that aren't cleaned up until the browser shuts down. So for all practical purposes these are leaks, but they don't show up as leaks in most to our leak tools as they are cleaned up during the shutdown process before we measure leaks. These types of leaks are hard to measure as they are in fact freed. The best solution we have is looking at the memory-over-time graph to see if it goes up during browser usage. And as always, DBaron has written tools that can help us, see Jesse's blog post here: http://www.squarefree.com/2006/01/13/memory-leak-detection-tool/ / Jonas _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Boris Zbarsky
::
Rate this Message:
Mike Schroepfer wrote:
> Right like bfcache and others. Would be awesome to have a big switch to > flush all of these buffers explicitly. bfcache is an observer for the "memory-pressure" notification from the observer service. So are the device context font cache (not sure we use that, though), stringbundles, the jar implementation, the XBL service. We might want to make more things observer this notification. Then it's easy to have an extension, or the main chrome, flush all this stuff and measure the perf impact. -Boris _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
|
|
|
Re: Memory fragmentation?
by Boris Zbarsky
::
Rate this Message:
robert@... wrote:
> I'm very willing to believe that heap fragmentation --- or just plain > failure to ever release brk memory --- is a problem. I don't know > offhand of any tools to give us the information malloc_stats on Linux can apparently give some relevant information. See for example https://bugzilla.mozilla.org/show_bug.cgi?id=130157#c4 Note that in that case fragmentation (or possibly not releasing brk memory) is indeed a problem. -Boris _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Jonas Sicking-2
::
Rate this Message:
Boris Zbarsky wrote:
> robert@... wrote: >> I'm very willing to believe that heap fragmentation --- or just plain >> failure to ever release brk memory --- is a problem. I don't know >> offhand of any tools to give us the information > > malloc_stats on Linux can apparently give some relevant information. > See for example https://bugzilla.mozilla.org/show_bug.cgi?id=130157#c4 > > Note that in that case fragmentation (or possibly not releasing brk > memory) is indeed a problem. Should fragmentation really be a problem over time? I guess I might be expecting too much of the alloc implementation, but since we should be allocating the same sized data structures over and over again (nsGenericElement, nsCParserNode, nsVoidArray::Impl), shouldn't we be able to reuse the empty fragments that are created while freeing the same structures? At least over time, say 10-20 pages. / Jonas _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Boris Zbarsky
::
Rate this Message:
Jonas Sicking wrote:
>> Note that in that case fragmentation (or possibly not releasing brk >> memory) is indeed a problem. > > Should fragmentation really be a problem over time? Well. In that bug the "problem" is "top reports high memory usage". But even apart from that, it can be. For example, it can force pages to be paged in as we access them for just the one or two words in them that are still allocated, with the result that either we actually do use close to the amount of RAM top reports or that we keep paging things in and out. Fragmentation that means that entire pages are free is probably not a huge deal over time. I hope. Or was the question whether we should get ever-increasing memory use due to fragmentation? -Boris _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Robert O'Callahan-2
::
Rate this Message:
On Oct 19, 1:17 pm, Boris Zbarsky <bzbar...@...> wrote:
> Well. In that bug the "problem" is "top reports high memory usage". Which is typically what people complain about, too. "I closed all my pages and tabs and Firefox is still using 10GB of memory!" We may have free()ed almost all of it, but the library won't give it back to the OS. The issue of fragmentation increasing the page working set is a real issue too, I'm sure. Rob _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Jonas Sicking-2
::
Rate this Message:
robert@... wrote:
> On Oct 19, 1:17 pm, Boris Zbarsky <bzbar...@...> wrote: >> Well. In that bug the "problem" is "top reports high memory usage". > > Which is typically what people complain about, too. "I closed all my > pages and tabs and Firefox is still using 10GB of memory!" We may have > free()ed almost all of it, but the library won't give it back to the > OS. One that is more of a perception problem, rather than a real technical one. > The issue of fragmentation increasing the page working set is a real > issue too, I'm sure. yeah, definitely, though one that shows up as slow performance rather than high memory use. This is definitely something we should try to address though. One interesting thing to note is that once we're using MMgc we will be allocating all our garbage collected objects using memory pools in the MMgc implementation. That has the upside that we have a little more control over things, but the downside that it'll be harder to have separate memory pools for, for example, each document. / Jonas _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Robert O'Callahan-2
::
Rate this Message:
On Oct 20, 11:04 am, Jonas Sicking <jo...@...> wrote:
> One interesting thing to note is that once we're using MMgc we will be > allocating all our garbage collected objects using memory pools in the > MMgc implementation. That has the upside that we have a little more > control over things, but the downside that it'll be harder to have > separate memory pools for, for example, each document. I wonder if we could implement a mostly-copying mode for MMgc. So if an object is not referenced by the stack, but and is referenced from the heap via only write-barriered pointers that we know about, we could move it. Rob _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Dan Mosedale
::
Rate this Message:
Jonas Sicking wrote:
> robert@... wrote: >> On Oct 19, 1:17 pm, Boris Zbarsky <bzbar...@...> wrote: >>> Well. In that bug the "problem" is "top reports high memory usage". >> >> Which is typically what people complain about, too. "I closed all my >> pages and tabs and Firefox is still using 10GB of memory!" We may have >> free()ed almost all of it, but the library won't give it back to the >> OS. > > One that is more of a perception problem, rather than a real technical one. Doesn't that depend on whether we're talking about RSIZE or VSIZE here, since (at least the Mac impl of) top displays both? VSIZE would certainly be a perceptual problem, but if we have lots of mostly-but-not-entirely-freed RSIZE pages hanging around, other apps are prevented from using them, which would be more real. Dan _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by schrep-2
::
Rate this Message:
Boris Zbarsky wrote:
> robert@... wrote: >> I'm very willing to believe that heap fragmentation --- or just plain >> failure to ever release brk memory --- is a problem. I don't know >> offhand of any tools to give us the information > > malloc_stats on Linux can apparently give some relevant information. > See for example https://bugzilla.mozilla.org/show_bug.cgi?id=130157#c4 > > Note that in that case fragmentation (or possibly not releasing brk > memory) is indeed a problem. > > -Boris > Hmm - that's awesome. Easy to get current data and/or data on windows? Mike _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by schrep-2
::
Rate this Message:
robert@... wrote:
> On Oct 19, 1:17 pm, Boris Zbarsky <bzbar...@...> wrote: >> Well. In that bug the "problem" is "top reports high memory usage". > > Which is typically what people complain about, too. "I closed all my > pages and tabs and Firefox is still using 10GB of memory!" We may have > free()ed almost all of it, but the library won't give it back to the > OS. > yep - really big perceptual problem. Just hitting spacbar in google reader can start to chew through ram... Closing all tabs and having a very different memory number than when you first start is very hard to explain. As I asked on another reply is there an easy way to get modern numbers on windows for this? Cheers, Schrep _______________________________________________ dev-performance mailing list dev-performance@... https://lists.mozilla.org/listinfo/dev-performance |
|
|
Re: Memory fragmentation?
by Stuart Parmenter-3
::
Rate this Message:
On Oct 26, 7:25 pm, Mike Schroepfer <sch...@...> wrote:
> rob...@... wrote: > > On Oct 19, 1:17 pm, Boris Zbarsky <bzbar...@...> wrote: > >> Well. In that bug the "problem" is "top reports high memory usage". > > > Which is typically what people complain about, too. "I closed all my > > pages and tabs and Firefox is still using 10GB of memory!" We may have > > free()ed almost all of it, but the library won't give it ba |
