Crash fread(...) failed (only read 0) (Cannot allocate memory) during deepFlatten with MLton 20070826

[Nicolas Bertolotti-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hello,

 

I am trying to investigate the following issue:

« 

         ssa2Simplify starting

            typeCheck starting

               checkScopes starting

               checkScopes finished in 0.82 + 0.00 (0% GC)

            typeCheck finished in 1.18 + 0.00 (0% GC)

            num functions 3141

            num blocks 170718

            num statements 230643

            deepFlatten starting

eof

fread (0x574e5000, 1, 2000101560, _) failed (only read 0).

 (Cannot allocate memory)

« 

 

It occurs when I build my application using a machine with 2061152k of RAM and 6859712k of swap (I have another machine with 4Gb of RAM on which it compiles fine). The crash appears while the RAM is full and the process swaps … but there is still lot of swap space available when it crashes.

 

Looking at the code, I could not figure how this could happen.

 

Any help would be appreciated.

 

Thanks in advance

 

Nicolas

 

P.S. : I attached the full log with option –verbose 3. 

 

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Fri, 18 Apr 2008, Nicolas Bertolotti wrote:
> eof
> fread (0x574e5000, 1, 2000101560, _) failed (only read 0).
>  (Cannot allocate memory)
>
> Looking at the code, I could not figure how this could happen.

It occurs when the runtime system decides to page the heap to disk in
order to free up virtual memory to reallocate the heap at a larger size.
It will do this if the desired heap size is approaching the physical
memory limit, even if there is plenty of swap space available on the
machine.

There is a bug in <src>/runtime/platform/diskBack.unix.c, introduced on
the x86_64-branch by SVN revision 4642 on 20060711.  That revision
switched to ANSI C stdio functions, but dropped the lseek in
diskBack_read.  This means that, when reading back the heap, it begins to
read from the temporary file with the file position at the end of the
written heap; that explains why fread failed with EOF.


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Sun, 20 Apr 2008, Matthew Fluet wrote:

> On Fri, 18 Apr 2008, Nicolas Bertolotti wrote:
>>  eof
>>  fread (0x574e5000, 1, 2000101560, _) failed (only read 0).
>>   (Cannot allocate memory)
>>
>>  Looking at the code, I could not figure how this could happen.
>
> It occurs when the runtime system decides to page the heap to disk in order
> to free up virtual memory to reallocate the heap at a larger size.
> It will do this if the desired heap size is approaching the physical memory
> limit, even if there is plenty of swap space available on the machine.
>
> There is a bug in <src>/runtime/platform/diskBack.unix.c, introduced on the
> x86_64-branch by SVN revision 4642 on 20060711.  That revision switched to
> ANSI C stdio functions, but dropped the lseek in diskBack_read.  This means
> that, when reading back the heap, it begins to read from the temporary file
> with the file position at the end of the written heap; that explains why
> fread failed with EOF.

I believe that the bug is now fixed by SVN r6596:
   http://mlton.org/cgi-bin/viewsvn.cgi?rev=6596&view=rev
You should be able to apply the diffs of that revision to the 20070826
sources, rebuild the compiler, and eliminate the compile abort.

If, on the other hand, you rebuild the compiler from SVN trunk HEAD, then
you should add "@MLton may-page-heap true" to the flags passed to mlton.
Because of the security and 'least-surprise' concerns of dumping 2+
gigabytes of heap to the disk during the execution of a program, the
runtime defaults to not paging the heap to disk, and instead will report
out-of-memory and terminate.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Sun, 20 Apr 2008, Matthew Fluet wrote:
> It occurs when the runtime system decides to page the heap to disk in order
> to free up virtual memory to reallocate the heap at a larger size.
> It will do this if the desired heap size is approaching the physical memory
> limit, even if there is plenty of swap space available on the machine.

Actually, this isn't right.  The runtime will attempt to page to disk only
if creating a heap of the minimum size cannot be satisfied with the
existing heap resident.

It is curious that your program compiles without paging the heap to disk
on the 4G machine, but not on the 2G machine, since the virtual memory
address space should be the same on the two machines.  That is, the amount
of physical memory shouldn't affect whether the virtual memory system is
able to satisfy the request.

Of course, starting off with different physical memory sizes means that
different heap sizes will be realized by the compile on the two machines,
so the point at which the paging occurs on the 2G machine may not
correspond to any GC on the 4G machine.


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

Thanks for the answer.

I applied the patch and tried to rebuild.

Unfortunately, here is what I get now:
            deepFlatten starting
/bin/sh: line 33: 26073 Segmentation fault      ./lock-exec.x86-linux /tmp/MLton-build.lock /mathworks/GNB/hub/pst/build-tools-R2008b-testing/bin/mlton -cc-opt -D_UNIX -align 4 -verbose 2 -const 'TextIO.bufSize 65536' -const 'MLton.detectOverflow false' -runtime no-load-world -native-split 100000 -cc-opt -s -native-optimize 3 -cc-opt -O2 -cc-opt -funroll-loops -cc-opt -ffast-math -cc-opt -fomit-frame-pointer -cc-opt -fno-exceptions -cc-opt -fexpensive-optimizations -cc-opt -freg-struct-return -cc-opt -unroll-all-loops -target i686-pc-linux -link-opt -L. -link-opt -lz -output iabc-dfa.cx86-linux sources.mlb MLton_ffi.c $EXTRA_C_FILES

I am still investigating but you may know what's going on.

Nicolas

> -----Original Message-----
> From: Matthew Fluet [mailto:fluet@...]
> Sent: Monday, April 21, 2008 2:00 AM
> To: Matthew Fluet
> Cc: Nicolas Bertolotti; mlton@...
> Subject: Re: [MLton] Crash fread(...) failed (only read 0) (Cannot
> allocate memory) during deepFlatten with MLton 20070826
>
> On Sun, 20 Apr 2008, Matthew Fluet wrote:
> > It occurs when the runtime system decides to page the heap to disk in
> order
> > to free up virtual memory to reallocate the heap at a larger size.
> > It will do this if the desired heap size is approaching the physical
> memory
> > limit, even if there is plenty of swap space available on the machine.
>
> Actually, this isn't right.  The runtime will attempt to page to disk only
> if creating a heap of the minimum size cannot be satisfied with the
> existing heap resident.
>
> It is curious that your program compiles without paging the heap to disk
> on the 4G machine, but not on the 2G machine, since the virtual memory
> address space should be the same on the two machines.  That is, the amount
> of physical memory shouldn't affect whether the virtual memory system is
> able to satisfy the request.
>
> Of course, starting off with different physical memory sizes means that
> different heap sizes will be realized by the compile on the two machines,
> so the point at which the paging occurs on the 2G machine may not
> correspond to any GC on the 4G machine.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

It seems that the segmentation fault actually corresponds to a different bug.

Here is the end of the log I get using the "gc-messages" flag:
...
[GC: Reading heap at 0x563cb000 of size 1,746,532,092 bytes from disk.]
[GC: Translating heap at 0x55719000 of size 0 bytes from 0x563cb000.]
[GC: Finished gc #160; time 28,789 ms,]
[GC:    heap at 0x55719000 of size 1,760,493,568 bytes,]
[GC:    with nursery of size 880,246,784 bytes (50.0% of heap),]
[GC:    and old-gen of size 0 bytes (0.0% of heap).]
/bin/sh: line 33: 25040 Segmentation fault
...

Still investigating ...

Nicolas

> -----Original Message-----
> From: mlton-bounces@... [mailto:mlton-bounces@...] On Behalf
> Of Nicolas Bertolotti
> Sent: Tuesday, April 22, 2008 1:40 AM
> To: Matthew Fluet
> Cc: mlton@...
> Subject: RE: [MLton] Crash fread(...) failed (only read 0) (Cannot
> allocate memory) during deepFlatten with MLton 20070826
>
> Thanks for the answer.
>
> I applied the patch and tried to rebuild.
>
> Unfortunately, here is what I get now:
>             deepFlatten starting
> /bin/sh: line 33: 26073 Segmentation fault      ./lock-exec.x86-linux
> /tmp/MLton-build.lock /mathworks/GNB/hub/pst/build-tools-R2008b-
> testing/bin/mlton -cc-opt -D_UNIX -align 4 -verbose 2 -const
> 'TextIO.bufSize 65536' -const 'MLton.detectOverflow false' -runtime no-
> load-world -native-split 100000 -cc-opt -s -native-optimize 3 -cc-opt -O2
> -cc-opt -funroll-loops -cc-opt -ffast-math -cc-opt -fomit-frame-pointer -
> cc-opt -fno-exceptions -cc-opt -fexpensive-optimizations -cc-opt -freg-
> struct-return -cc-opt -unroll-all-loops -target i686-pc-linux -link-opt -
> L. -link-opt -lz -output iabc-dfa.cx86-linux sources.mlb MLton_ffi.c
> $EXTRA_C_FILES
>
> I am still investigating but you may know what's going on.
>
> Nicolas
>
> > -----Original Message-----
> > From: Matthew Fluet [mailto:fluet@...]
> > Sent: Monday, April 21, 2008 2:00 AM
> > To: Matthew Fluet
> > Cc: Nicolas Bertolotti; mlton@...
> > Subject: Re: [MLton] Crash fread(...) failed (only read 0) (Cannot
> > allocate memory) during deepFlatten with MLton 20070826
> >
> > On Sun, 20 Apr 2008, Matthew Fluet wrote:
> > > It occurs when the runtime system decides to page the heap to disk in
> > order
> > > to free up virtual memory to reallocate the heap at a larger size.
> > > It will do this if the desired heap size is approaching the physical
> > memory
> > > limit, even if there is plenty of swap space available on the machine.
> >
> > Actually, this isn't right.  The runtime will attempt to page to disk
> only
> > if creating a heap of the minimum size cannot be satisfied with the
> > existing heap resident.
> >
> > It is curious that your program compiles without paging the heap to disk
> > on the 4G machine, but not on the 2G machine, since the virtual memory
> > address space should be the same on the two machines.  That is, the
> amount
> > of physical memory shouldn't affect whether the virtual memory system is
> > able to satisfy the request.
> >
> > Of course, starting off with different physical memory sizes means that
> > different heap sizes will be realized by the compile on the two
> machines,
> > so the point at which the paging occurs on the 2G machine may not
> > correspond to any GC on the 4G machine.
>
>
> _______________________________________________
> MLton mailing list
> MLton@...
> http://mlton.org/mailman/listinfo/mlton

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

There was actually another bug in heap.c. Here is an additional patch to fix it:
--- mlton/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
+++ mltonp1/runtime/gc/heap.c 2008-04-22 16:24:50.000000000 +0200
@@ -337,6 +337,7 @@
       }
       GC_diskBack_read (data, curHeapp->start, size);
       GC_diskBack_close (data);
+      curHeapp->oldGenSize = size;
     } else {
       GC_diskBack_close (data);
       if (s->controls.messages)

Unfortunately, now, I get the following error:
            deepFlatten starting
Out of memory.  Unable to allocate 1,686,393,408 bytes.

... which, I guess, is a different problem.

Nicolas

> -----Original Message-----
> From: Nicolas Bertolotti
> Sent: Tuesday, April 22, 2008 4:13 PM
> To: Nicolas Bertolotti; Matthew Fluet
> Cc: mlton@...
> Subject: RE: [MLton] Crash fread(...) failed (only read 0) (Cannot
> allocate memory) during deepFlatten with MLton 20070826
>
> It seems that the segmentation fault actually corresponds to a different
> bug.
>
> Here is the end of the log I get using the "gc-messages" flag:
> ...
> [GC: Reading heap at 0x563cb000 of size 1,746,532,092 bytes from disk.]
> [GC: Translating heap at 0x55719000 of size 0 bytes from 0x563cb000.]
> [GC: Finished gc #160; time 28,789 ms,]
> [GC:    heap at 0x55719000 of size 1,760,493,568 bytes,]
> [GC:    with nursery of size 880,246,784 bytes (50.0% of heap),]
> [GC:    and old-gen of size 0 bytes (0.0% of heap).]
> /bin/sh: line 33: 25040 Segmentation fault
> ...
>
> Still investigating ...
>
> Nicolas
>
> > -----Original Message-----
> > From: mlton-bounces@... [mailto:mlton-bounces@...] On Behalf
> > Of Nicolas Bertolotti
> > Sent: Tuesday, April 22, 2008 1:40 AM
> > To: Matthew Fluet
> > Cc: mlton@...
> > Subject: RE: [MLton] Crash fread(...) failed (only read 0) (Cannot
> > allocate memory) during deepFlatten with MLton 20070826
> >
> > Thanks for the answer.
> >
> > I applied the patch and tried to rebuild.
> >
> > Unfortunately, here is what I get now:
> >             deepFlatten starting
> > /bin/sh: line 33: 26073 Segmentation fault      ./lock-exec.x86-linux
> > /tmp/MLton-build.lock /mathworks/GNB/hub/pst/build-tools-R2008b-
> > testing/bin/mlton -cc-opt -D_UNIX -align 4 -verbose 2 -const
> > 'TextIO.bufSize 65536' -const 'MLton.detectOverflow false' -runtime no-
> > load-world -native-split 100000 -cc-opt -s -native-optimize 3 -cc-opt -
> O2
> > -cc-opt -funroll-loops -cc-opt -ffast-math -cc-opt -fomit-frame-pointer
> -
> > cc-opt -fno-exceptions -cc-opt -fexpensive-optimizations -cc-opt -freg-
> > struct-return -cc-opt -unroll-all-loops -target i686-pc-linux -link-opt
> -
> > L. -link-opt -lz -output iabc-dfa.cx86-linux sources.mlb MLton_ffi.c
> > $EXTRA_C_FILES
> >
> > I am still investigating but you may know what's going on.
> >
> > Nicolas
> >
> > > -----Original Message-----
> > > From: Matthew Fluet [mailto:fluet@...]
> > > Sent: Monday, April 21, 2008 2:00 AM
> > > To: Matthew Fluet
> > > Cc: Nicolas Bertolotti; mlton@...
> > > Subject: Re: [MLton] Crash fread(...) failed (only read 0) (Cannot
> > > allocate memory) during deepFlatten with MLton 20070826
> > >
> > > On Sun, 20 Apr 2008, Matthew Fluet wrote:
> > > > It occurs when the runtime system decides to page the heap to disk
> in
> > > order
> > > > to free up virtual memory to reallocate the heap at a larger size.
> > > > It will do this if the desired heap size is approaching the physical
> > > memory
> > > > limit, even if there is plenty of swap space available on the
> machine.
> > >
> > > Actually, this isn't right.  The runtime will attempt to page to disk
> > only
> > > if creating a heap of the minimum size cannot be satisfied with the
> > > existing heap resident.
> > >
> > > It is curious that your program compiles without paging the heap to
> disk
> > > on the 4G machine, but not on the 2G machine, since the virtual memory
> > > address space should be the same on the two machines.  That is, the
> > amount
> > > of physical memory shouldn't affect whether the virtual memory system
> is
> > > able to satisfy the request.
> > >
> > > Of course, starting off with different physical memory sizes means
> that
> > > different heap sizes will be realized by the compile on the two
> > machines,
> > > so the point at which the paging occurs on the 2G machine may not
> > > correspond to any GC on the 4G machine.
> >
> >
> > _______________________________________________
> > MLton mailing list
> > MLton@...
> > http://mlton.org/mailman/listinfo/mlton

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Tue, 22 Apr 2008, Nicolas Bertolotti wrote:

> There was actually another bug in heap.c. Here is an additional patch to fix it:
> --- mlton/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
> +++ mltonp1/runtime/gc/heap.c 2008-04-22 16:24:50.000000000 +0200
> @@ -337,6 +337,7 @@
>       }
>       GC_diskBack_read (data, curHeapp->start, size);
>       GC_diskBack_close (data);
> +      curHeapp->oldGenSize = size;
>     } else {
>       GC_diskBack_close (data);
>       if (s->controls.messages)

Yes, I was just coming to that exact conclusion.

> Unfortunately, now, I get the following error:
>            deepFlatten starting
> Out of memory.  Unable to allocate 1,686,393,408 bytes.
>
> ... which, I guess, is a different problem.

It is a different problem, in the sense that you succeeded (at least
once) in writing the heap to disk, allocating a larger heap, and reading
the heap from disk.  Getting an out of memory error suggests that even
after writing the heap to disk and releasing it, the operating system was
unable to mmap 1.6G.  That seems strange, even on a 32bit system, where
the virtual address space is 4G, though I suppose it could happen with
sufficient fragmentation.

If you compile with @MLton gc-messages --, you should get a more detailed
memory listing of what memory is reserved, which might indicate whether
there is some extra memory being held.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

> It is a different problem, in the sense that you succeeded (at least
> once) in writing the heap to disk, allocating a larger heap, and reading
> the heap from disk.  Getting an out of memory error suggests that even
> after writing the heap to disk and releasing it, the operating system was
> unable to mmap 1.6G.  That seems strange, even on a 32bit system, where
> the virtual address space is 4G, though I suppose it could happen with
> sufficient fragmentation.
>
> If you compile with @MLton gc-messages --, you should get a more detailed
> memory listing of what memory is reserved, which might indicate whether
> there is some extra memory being held.

It is exactly what I am doing right now...

When the failure occurs, there is still a lot of swap space available ... and, as you mentioned, the virtual address space is the same on the machine with 4Gb of RAM and the 2Gb one.

I just noticed that the failure occurs while the GC tries to allocate a new heap with a "minSize" value that is actually smaller than the size of the previous heap (that has just been released).

I have applied the following patch (just to give a try):
--- mlton/runtime/gc/heap.c     2008-04-22 18:27:34.000000000 +0200
+++ mltonp1/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
@@ -208,9 +208,6 @@
                uintmaxToCommaString (backoff),
                uintmaxToCommaString (minSize));
     }
+    /* Make sure we always try to allocate at least 'minSize' bytes. */
+    if (h->size > minSize && (h->size - backoff) < minSize)
+      backoff = h->size - minSize;
   }
   h->size = 0;
   return FALSE;

Nicolas


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

I finally applied the following patch in order to deal with the page size aspect:
--- mlton/runtime/gc/heap.c        2008-04-11 19:12:58.000000000 +0200
+++ mltonp1/runtime/gc/heap.c      2008-04-22 19:28:35.000000000 +0200
@@ -157,6 +157,7 @@
   assert (isHeapInit (h));
   if (desiredSize < minSize)
     desiredSize = minSize;
+  minSize = align (minSize, s->sysvals.pageSize);
   desiredSize = align (desiredSize, s->sysvals.pageSize);
   assert (0 == h->size and NULL == h->start);
   backoff = (desiredSize - minSize) / 20;
@@ -208,6 +209,9 @@
                uintmaxToCommaString (backoff),
                uintmaxToCommaString (minSize));
     }
+    /* Make sure we always try to allocate at least 'minSize' bytes. */
+    if (h->size > minSize && (h->size - backoff) < minSize)
+      backoff = h->size - minSize;
   }
   h->size = 0;
   return FALSE;

Now, the compilation succeeds ... but it is very slow. A lot of time seems to be spent on GC even while there is about 300 Mb of available memory that could be allocated to get a bigger heap.

Anyway, I am not an expert on the subject and I don't know much about MLton's GC.

I have just applied your latest patch about card cross map and I'll try to give you some feedback.

Nicolas

> -----Original Message-----
> From: Matthew Fluet [mailto:fluet@...]
> Sent: Tuesday, April 22, 2008 8:13 PM
> To: Matthew Fluet
> Cc: Nicolas Bertolotti; mlton@...
> Subject: RE: [MLton] Crash fread(...) failed (only read 0) (Cannot
> allocate memory) during deepFlatten with MLton 20070826
>
> On Tue, 22 Apr 2008, Matthew Fluet wrote:
> > On Tue, 22 Apr 2008, Nicolas Bertolotti wrote:
> >>  There was actually another bug in heap.c. Here is an additional patch
> to
> >>  fix it:
> >>  --- mlton/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
> >>  +++ mltonp1/runtime/gc/heap.c 2008-04-22 16:24:50.000000000 +0200
> >>  @@ -337,6 +337,7 @@
> >>        }
> >>        GC_diskBack_read (data, curHeapp->start, size);
> >>        GC_diskBack_close (data);
> >>  +      curHeapp->oldGenSize = size;
> >>      } else {
> >>        GC_diskBack_close (data);
> >>        if (s->controls.messages)
> >
> > Yes, I was just coming to that exact conclusion.
> >
> >>  Unfortunately, now, I get the following error:
> >>             deepFlatten starting
> >>  Out of memory.  Unable to allocate 1,686,393,408 bytes.
> >>
> >>  ... which, I guess, is a different problem.
> >
> > It is a different problem, in the sense that you succeeded (at least
> once) in
> > writing the heap to disk, allocating a larger heap, and reading the heap
> from
> > disk.  Getting an out of memory error suggests that even after writing
> the
> > heap to disk and releasing it, the operating system was unable to mmap
> 1.6G.
> > That seems strange, even on a 32bit system, where the virtual address
> space
> > is 4G, though I suppose it could happen with sufficient fragmentation.
> >
> > If you compile with @MLton gc-messages --, you should get a more
> detailed
> > memory listing of what memory is reserved, which might indicate whether
> there
> > is some extra memory being held.
>
> One thing I notice is that when writing the heap to disk, we do not
> release the card/cross maps.  Although with a 1.6G heap, these are only
> 12.8M, if they happen to fall in the middle of the virtual address space,
> then it may not be possible to find the requested 1.6G heap.
>
> You could try the attached patch, which releases the card/cross maps after
> writing the heap to disk.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Tue, 22 Apr 2008, Nicolas Bertolotti wrote:
> When the failure occurs, there is still a lot of swap space available
> ... and, as you mentioned, the virtual address space is the same on the
> machine with 4Gb of RAM and the 2Gb one.

I could be that the different amounts of physical memory result in the
operating system putting things at different virtual memory addresses.

> I just noticed that the failure occurs while the GC tries to allocate a
> new heap with a "minSize" value that is actually smaller than the size
> of the previous heap (that has just been released).

Good point; one wouldn't expect a heap resizing to fail if the minimum
size is smaller than the size of the heap in hand.

> I have applied the following patch (just to give a try):
> --- mlton/runtime/gc/heap.c     2008-04-22 18:27:34.000000000 +0200
> +++ mltonp1/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
> @@ -208,9 +208,6 @@
>                uintmaxToCommaString (backoff),
>                uintmaxToCommaString (minSize));
>     }
> +    /* Make sure we always try to allocate at least 'minSize' bytes. */
> +    if (h->size > minSize && (h->size - backoff) < minSize)
> +      backoff = h->size - minSize;
>   }
>   h->size = 0;
>   return FALSE;

I see: this causes the last round of the backoff to always try the minimum
size.  Since minSize isn't necessarily pageSize aligned, you might want to
tweak the conditions so that that mmap isn't attempted with a non-pageSize
aligned value.


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Tue, 22 Apr 2008, Matthew Fluet wrote:

> On Tue, 22 Apr 2008, Nicolas Bertolotti wrote:
>>  There was actually another bug in heap.c. Here is an additional patch to
>>  fix it:
>>  --- mlton/runtime/gc/heap.c   2008-04-11 19:12:58.000000000 +0200
>>  +++ mltonp1/runtime/gc/heap.c 2008-04-22 16:24:50.000000000 +0200
>>  @@ -337,6 +337,7 @@
>>        }
>>        GC_diskBack_read (data, curHeapp->start, size);
>>        GC_diskBack_close (data);
>>  +      curHeapp->oldGenSize = size;
>>      } else {
>>        GC_diskBack_close (data);
>>        if (s->controls.messages)
>
> Yes, I was just coming to that exact conclusion.
>
>>  Unfortunately, now, I get the following error:
>>             deepFlatten starting
>>  Out of memory.  Unable to allocate 1,686,393,408 bytes.
>>
>>  ... which, I guess, is a different problem.
>
> It is a different problem, in the sense that you succeeded (at least once) in
> writing the heap to disk, allocating a larger heap, and reading the heap from
> disk.  Getting an out of memory error suggests that even after writing the
> heap to disk and releasing it, the operating system was unable to mmap 1.6G.
> That seems strange, even on a 32bit system, where the virtual address space
> is 4G, though I suppose it could happen with sufficient fragmentation.
>
> If you compile with @MLton gc-messages --, you should get a more detailed
> memory listing of what memory is reserved, which might indicate whether there
> is some extra memory being held.

One thing I notice is that when writing the heap to disk, we do not
release the card/cross maps.  Although with a 1.6G heap, these are only
12.8M, if they happen to fall in the middle of the virtual address space,
then it may not be possible to find the requested 1.6G heap.

You could try the attached patch, which releases the card/cross maps after
writing the heap to disk.
diff --git a/runtime/gc/done.c b/runtime/gc/done.c
index a7c1774..4838b56 100644
--- a/runtime/gc/done.c
+++ b/runtime/gc/done.c
@@ -93,4 +93,11 @@ void GC_done (GC_state s) {
   }
   releaseHeap (s, &s->heap);
   releaseHeap (s, &s->secondaryHeap);
+  if (s->mutatorMarksCards) {
+    releaseCardMapAndCrossMap (s,
+                               s->generationalMaps.cardMap,
+                               s->generationalMaps.cardMapLength * CARD_MAP_ELEM_SIZE,
+                               s->generationalMaps.crossMap,
+                               s->generationalMaps.crossMapLength * CROSS_MAP_ELEM_SIZE);
+  }
 }
diff --git a/runtime/gc/generational.c b/runtime/gc/generational.c
index 93124bc..f2b8f4d 100644
--- a/runtime/gc/generational.c
+++ b/runtime/gc/generational.c
@@ -147,13 +147,15 @@ void createCardMapAndCrossMap (GC_state s) {
   s->generationalMaps.crossMapLength = crossMapLength;
 
   totalMapSize = cardMapSize + crossMapSize;
-  if (DEBUG_MEM)
-    fprintf (stderr, "Creating card/cross map of size %s\n",
-             uintmaxToCommaString(totalMapSize));
   s->generationalMaps.cardMap =
     GC_mmapAnon_safe (NULL, totalMapSize);
   s->generationalMaps.crossMap =
     (s->generationalMaps.cardMap + (cardMapSize / CARD_MAP_ELEM_SIZE));
+  if (DEBUG_MEM or s->controls.messages)
+    fprintf (stderr,
+             "[GC: Created card/cross map at "FMTPTR" of size %s bytes.]\n",
+             (uintptr_t)(s->generationalMaps.cardMap),
+             uintmaxToCommaString(totalMapSize));
   if (DEBUG_CARD_MARKING)
     fprintf (stderr, "cardMap = "FMTPTR"  crossMap = "FMTPTR"\n",
              (uintptr_t)s->generationalMaps.cardMap,
@@ -163,6 +165,24 @@ void createCardMapAndCrossMap (GC_state s) {
   clearCrossMap (s);
 }
 
+void releaseCardMapAndCrossMap (GC_state s,
+                                GC_cardMap cardMap,
+                                size_t cardMapSize,
+                                __attribute__ ((unused)) GC_crossMap crossMap,
+                                size_t crossMapSize) {
+
+  size_t totalMapSize;
+
+  assert (crossMap == cardMap + (cardMapSize / CARD_MAP_ELEM_SIZE));
+  totalMapSize = cardMapSize + crossMapSize;
+  if (DEBUG_MEM or s->controls.messages)
+    fprintf (stderr,
+             "[GC: Releasing card/cross map at "FMTPTR" of size %s bytes.]\n",
+             (uintptr_t)cardMap,
+             uintmaxToCommaString(totalMapSize));
+  GC_release (cardMap, totalMapSize);
+}
+
 void resizeCardMapAndCrossMap (GC_state s) {
   if (s->mutatorMarksCards
       and (s->generationalMaps.cardMapLength * CARD_MAP_ELEM_SIZE)
@@ -180,9 +200,7 @@ void resizeCardMapAndCrossMap (GC_state s) {
     GC_memcpy ((pointer)oldCrossMap, (pointer)s->generationalMaps.crossMap,
                min (s->generationalMaps.crossMapLength * CROSS_MAP_ELEM_SIZE,
                     oldCrossMapSize));
-    if (DEBUG_MEM)
-      fprintf (stderr, "Releasing card/cross map.\n");
-    GC_release (oldCardMap, oldCardMapSize + oldCrossMapSize);
+    releaseCardMapAndCrossMap (s, oldCardMap, oldCardMapSize, oldCrossMap, oldCrossMapSize);
   }
 }
 
diff --git a/runtime/gc/generational.h b/runtime/gc/generational.h
index 45e30a1..d16d84f 100644
--- a/runtime/gc/generational.h
+++ b/runtime/gc/generational.h
@@ -77,6 +77,9 @@ static inline pointer getCrossMapCardStart (GC_state s, pointer p);
 static inline void clearCardMap (GC_state s);
 static inline void clearCrossMap (GC_state s);
 static void createCardMapAndCrossMap (GC_state s);
+static void releaseCardMapAndCrossMap (GC_state s,
+                                       GC_cardMap cardMap, size_t cardMapSize,
+                                       GC_crossMap crossMap, size_t crossMapSize);
 static void resizeCardMapAndCrossMap (GC_state s);
 
 #if ASSERT
diff --git a/runtime/gc/heap.c b/runtime/gc/heap.c
index c042b46..365418a 100644
--- a/runtime/gc/heap.c
+++ b/runtime/gc/heap.c
@@ -337,6 +337,13 @@ copy:
     }
     data = GC_diskBack_write (orig, size);
     releaseHeap (s, curHeapp);
+    if (s->mutatorMarksCards) {
+      releaseCardMapAndCrossMap (s,
+                                 s->generationalMaps.cardMap,
+                                 s->generationalMaps.cardMapLength * CARD_MAP_ELEM_SIZE,
+                                 s->generationalMaps.crossMap,
+                                 s->generationalMaps.crossMapLength * CROSS_MAP_ELEM_SIZE);
+    }
     if (createHeap (s, curHeapp, desiredSize, minSize)) {
       if (DEBUG or s->controls.messages) {
         fprintf (stderr,
@@ -347,6 +354,10 @@ copy:
       GC_diskBack_read (data, curHeapp->start, size);
       GC_diskBack_close (data);
       curHeapp->oldGenSize = size;
+      if (s->mutatorMarksCards) {
+        createCardMapAndCrossMap (s);
+        updateCrossMap (s);
+      }
     } else {
       GC_diskBack_close (data);
       if (s->controls.messages)

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

> > One thing I notice is that when writing the heap to disk, we do not
> > release the card/cross maps.  Although with a 1.6G heap, these are only
> > 12.8M, if they happen to fall in the middle of the virtual address
> space,
> > then it may not be possible to find the requested 1.6G heap.
> >
> > You could try the attached patch, which releases the card/cross maps
> after
> > writing the heap to disk.

I could notice that, when the heap is copied to the disk and createHeap() is called to allocate a new heap, there are multiple iterations of the "backoff" loop before mmap() succeeds. As a consequence, the newly allocated heap is not as big as what the GC algorithm would normally expect.

Using your latest patch (still combined with mine), I could notice that we don't loop anymore and the new heap size corresponds to what the GC algorithm expects (Then, I guess it will also make it possible to run more instructions without entering GC again).

So, it really seems that releasing the card/cross map before allocating the heap is a good way to reduce memory fragmentation.

It might be interesting to release and rebuild the card/cross map in other circumstances in order to achieve better performances:
- if remapHeap() fails, we could retry after releasing the card/cross map before deciding to create a new heap and copy the existing one.
- if createHeap() fails, we could retry after releasing the card/cross map before deciding to back up the existing heap to the disk.

... it depends on what the cost of rebuilding the card/cross map is (for the 2nd option, I guess it is certainly much less than transferring 1.5 Gb to disk).

What do you think?

Nicolas

P.S.: the compilation no longer crashes on my 2 Gb machine so it seems that the initial issue has now been fully fixed.

Unfortunately, the compilation swaps a lot (really a lot) during deepFlatten and I'm afraid it has become impossible to build the product using a machine that has less than 3 Gb of RAM. Any idea about what we could do to limit the memory consumption during this phase?


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton