Working: MLton/Win64 toolchain

[Wesley W. Terpstra-5]

I'm happy to announce that MLton now works perfectly on 64-bit windows using the native codegen. Please give it a spin and report any problems. It has only been tested on Win2003/64 and I would be very interested in hearing reports about XP64 and Vista64.

A complete toolchain (no additional software required) can be found at: http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If someone could move this to the Experimental page on the wiki, I would appreciate it (I don't have write permissions there). Be warned that the toolchain is a 180MB download.

To use the toolchain, uncompress the zip file wherever you like. The contained MLton\bin\mlton.bat should work from the windows shell. If you want to run the included 'msys.bat' to get a UNIX-like environment (make, sed, tar, etc), you must decompress the tarball to a location which does not include spaces in the path name. To rebuild MLton/svn, use 'make all-no-docs' in a checkout with the msys shell.

The toolchain includes:
* a 20080728 mingw-w64 toolchain
* gmp-mpfr mingw64 fork, 20080604
* msys 1.0.11pre, 20070119
* MLton r6698, 20080808

There remain problems with code generated by mingw-w64 gcc. The mingw toolchain included is actually 32 bit, cross-compiling to 64 in order to avoid problems with the native compiled linker. For this reason, I don't advise using the c-codegen. The bytecodegen and amdcodegen are fine as gcc has very little to do.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Hello,

 

This is great news. Thanks for the update.

 

I built a cross compiler from Linux to Win64 and tried to port my product using it.

 

So far, it seems that the pure SML code runs pretty fine on Win64 but there still a few issues elsewhere.

 

1. Spawn

 

I faced some troubles spawning child processes when the arguments contain some backslashes. Anyway, I am not sure this is related to the port as it also affects Win32 :

 

I could notice that a new cmdEscape() function has been recently added to fix a common issue with _spawnve() on Windows when the arguments contain some white spaces. This function duplicates the backslashes which appears to be incorrect.

 

I have applied the following patch in order to solve the issue :

--- basis-library/mlton/process.sml.old   2008-08-10 00:24:13.000000000 +0200

+++ basis-library/mlton/process.sml 2008-08-08 23:08:28.000000000 +0200

@@ -255,8 +255,7 @@

       fun cmdEscape y =

          concat [dquote,

                  String.translate

-(*                 (fn #"\"" => "\\\"" | #"\\" => "\\\\" | x => String.str x) y,*)

-                (fn #"\\" => "\\" | x => Char.toString x) y,

+                 (fn #"\"" => "\\\"" | #"\\" => "\\\\" | x => String.str x) y,

                  dquote]

 

       fun create (cmd, args, env, stdin, stdout, stderr) =

 

2. FFI

 

I am now facing a more complex issue with the foreign function interface. You will find attached a small example (30 lines of C code ; 2 lines of SML code) which reproduces the issue I am facing. In the example, when ffi_func() is called from SML code, it crashes (probably some kind of stack corruption) whereas it runs fine when it is executed from a C main().

 

I am stuck here so any idea is welcome.

 

Best regards

 

Nicolas

 

---

From: mlton-bounces@... [mailto:mlton-bounces@...] On Behalf Of Wesley W. Terpstra
Sent: Friday, August 08, 2008 8:06 PM
To: mlton@...; mlton-user@...
Subject: [MLton] Working: MLton/Win64 toolchain

 

I'm happy to announce that MLton now works perfectly on 64-bit windows using the native codegen. Please give it a spin and report any problems. It has only been tested on Win2003/64 and I would be very interested in hearing reports about XP64 and Vista64.

A complete toolchain (no additional software required) can be found at: http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If someone could move this to the Experimental page on the wiki, I would appreciate it (I don't have write permissions there). Be warned that the toolchain is a 180MB download.

To use the toolchain, uncompress the zip file wherever you like. The contained MLton\bin\mlton.bat should work from the windows shell. If you want to run the included 'msys.bat' to get a UNIX-like environment (make, sed, tar, etc), you must decompress the tarball to a location which does not include spaces in the path name. To rebuild MLton/svn, use 'make all-no-docs' in a checkout with the msys shell.

The toolchain includes:
* a 20080728 mingw-w64 toolchain
* gmp-mpfr mingw64 fork, 20080604
* msys 1.0.11pre, 20070119
* MLton r6698, 20080808

There remain problems with code generated by mingw-w64 gcc. The mingw toolchain included is actually 32 bit, cross-compiling to 64 in order to avoid problems with the native compiled linker. For this reason, I don't advise using the c-codegen. The bytecodegen and amdcodegen are fine as gcc has very little to do.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

It seems that some mail servers will reject the attached archive I previously sent. Here are the source files …

 

---

From: Nicolas Bertolotti
Sent: Monday, August 11, 2008 3:51 PM
To: 'Wesley W. Terpstra'; mlton@...; mlton-user@...
Subject: RE: [MLton] Working: MLton/Win64 toolchain

 

Hello,

 

This is great news. Thanks for the update.

 

I built a cross compiler from Linux to Win64 and tried to port my product using it.

 

So far, it seems that the pure SML code runs pretty fine on Win64 but there still a few issues elsewhere.

 

1. Spawn

 

I faced some troubles spawning child processes when the arguments contain some backslashes. Anyway, I am not sure this is related to the port as it also affects Win32 :

 

I could notice that a new cmdEscape() function has been recently added to fix a common issue with _spawnve() on Windows when the arguments contain some white spaces. This function duplicates the backslashes which appears to be incorrect.

 

I have applied the following patch in order to solve the issue :

--- basis-library/mlton/process.sml.old   2008-08-10 00:24:13.000000000 +0200

+++ basis-library/mlton/process.sml 2008-08-08 23:08:28.000000000 +0200

@@ -255,8 +255,7 @@

       fun cmdEscape y =

          concat [dquote,

                  String.translate

-(*                 (fn #"\"" => "\\\"" | #"\\" => "\\\\" | x => String.str x) y,*)

-                (fn #"\\" => "\\" | x => Char.toString x) y,

+                 (fn #"\"" => "\\\"" | #"\\" => "\\\\" | x => String.str x) y,

                  dquote]

 

       fun create (cmd, args, env, stdin, stdout, stderr) =

 

2. FFI

 

I am now facing a more complex issue with the foreign function interface. You will find attached a small example (30 lines of C code ; 2 lines of SML code) which reproduces the issue I am facing. In the example, when ffi_func() is called from SML code, it crashes (probably some kind of stack corruption) whereas it runs fine when it is executed from a C main().

 

I am stuck here so any idea is welcome.

 

Best regards

 

Nicolas

 

---

From: mlton-bounces@... [mailto:mlton-bounces@...] On Behalf Of Wesley W. Terpstra
Sent: Friday, August 08, 2008 8:06 PM
To: mlton@...; mlton-user@...
Subject: [MLton] Working: MLton/Win64 toolchain

 

I'm happy to announce that MLton now works perfectly on 64-bit windows using the native codegen. Please give it a spin and report any problems. It has only been tested on Win2003/64 and I would be very interested in hearing reports about XP64 and Vista64.

A complete toolchain (no additional software required) can be found at: http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If someone could move this to the Experimental page on the wiki, I would appreciate it (I don't have write permissions there). Be warned that the toolchain is a 180MB download.

To use the toolchain, uncompress the zip file wherever you like. The contained MLton\bin\mlton.bat should work from the windows shell. If you want to run the included 'msys.bat' to get a UNIX-like environment (make, sed, tar, etc), you must decompress the tarball to a location which does not include spaces in the path name. To rebuild MLton/svn, use 'make all-no-docs' in a checkout with the msys shell.

The toolchain includes:
* a 20080728 mingw-w64 toolchain
* gmp-mpfr mingw64 fork, 20080604
* msys 1.0.11pre, 20070119
* MLton r6698, 20080808

There remain problems with code generated by mingw-w64 gcc. The mingw toolchain included is actually 32 bit, cross-compiling to 64 in order to avoid problems with the native compiled linker. For this reason, I don't advise using the c-codegen. The bytecodegen and amdcodegen are fine as gcc has very little to do.

#include <stdio.h>

struct my_struct {
  void *p;
  unsigned char buf[16384];
};

void func(void *p) { }

extern void func2(void ** pp)
{
  struct my_struct s;

  if (pp == NULL) {
    func(&s.p);
  } else {
    s.p = *pp;
  }
  fprintf(stderr, "dummy2\n");
}

extern void ffi_func()
{
  func2(NULL);
  fprintf(stderr, "Test OK\n");
}

#ifdef C_MAIN
int main()
{
  ffi_func();
  return 0;
}
#endif



_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti <Nicolas.Bertolotti@...>

I built a cross compiler from Linux to Win64 and tried to port my product using it. So far, it seems that the pure SML code runs pretty fine on Win64 but there still a few issues elsewhere.

1. Spawn 

I faced some troubles spawning child processes when the arguments contain some backslashes. Anyway, I am not sure this is related to the port as it also affects Win32 :

I could notice that a new cmdEscape() function has been recently added to fix a common issue with _spawnve() on Windows when the arguments contain some white spaces. This function duplicates the backslashes which appears to be incorrect.

I'm looking into this. The fix is not as simple as your proposal. It seems some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/

2. FFI

I am now facing a more complex issue with the foreign function interface. You will find attached a small example (30 lines of C code ; 2 lines of SML code) which reproduces the issue I am facing. In the example, when ffi_func() is called from SML code, it crashes (probably some kind of stack corruption) whereas it runs fine when it is executed from a C main().

This isn't MLton's fault per-se. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg:
gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32
with main.c:

void ffi_func();
int main() { ffi_func(); return 0; }

I have seen this before. With win64 big allocations on the stack seem flaky. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct s as a stack-allocated variable. I've noticed that mingw also links to kernel32 on its own (now?). So you could edit the mlton script to not include -lkernel32, which also seems to fix it for some reason.

I'm rebuilding MLton to see whether all regressions pass with -lkernel32 removed from the link line. If it works (and I'll have to check win32 as well), I'll think about removing this from the mlton script by default.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti <Nicolas.Bertolotti@...>

I built a cross compiler from Linux to Win64 and tried to port my product using it. So far, it seems that the pure SML code runs pretty fine on Win64 but there still a few issues elsewhere.

1. Spawn 

I faced some troubles spawning child processes when the arguments contain some backslashes. Anyway, I am not sure this is related to the port as it also affects Win32 :

I could notice that a new cmdEscape() function has been recently added to fix a common issue with _spawnve() on Windows when the arguments contain some white spaces. This function duplicates the backslashes which appears to be incorrect.

I'm looking into this. The fix is not as simple as your proposal. It seems some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/

2. FFI

I am now facing a more complex issue with the foreign function interface. You will find attached a small example (30 lines of C code ; 2 lines of SML code) which reproduces the issue I am facing. In the example, when ffi_func() is called from SML code, it crashes (probably some kind of stack corruption) whereas it runs fine when it is executed from a C main().

This isn't MLton's fault per-se. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg:
gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32
with main.c:

void ffi_func();
int main() { ffi_func(); return 0; }

I have seen this before. With win64 big allocations on the stack seem flaky. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct s as a stack-allocated variable. I've noticed that mingw also links to kernel32 on its own (now?). So you could edit the mlton script to not include -lkernel32, which also seems to fix it for some reason.

I'm rebuilding MLton to see whether all regressions pass with -lkernel32 removed from the link line. If it works (and I'll have to check win32 as well), I'll think about removing this from the mlton script by default.

PS. I couldn't reply to you directly. Check your mail server...

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

On Mon, Aug 11, 2008 at 4:02 PM, Nicolas Bertolotti <Nicolas.Bertolotti@...>

I built a cross compiler from Linux to Win64 and tried to port my product using it. So far, it seems that the pure SML code runs pretty fine on Win64 but there still a few issues elsewhere.

1. Spawn 

I faced some troubles spawning child processes when the arguments contain some backslashes. Anyway, I am not sure this is related to the port as it also affects Win32 :

I could notice that a new cmdEscape() function has been recently added to fix a common issue with _spawnve() on Windows when the arguments contain some white spaces. This function duplicates the backslashes which appears to be incorrect.

I've fixed this in svn. The fix is not as simple as your proposal. Some \s need to be escaped, but not others. It also apparently differs between MinGW and cygwin. :-/

Please give the process.sml in svn/HEAD a whirl. Do you have cygwin also? I would like to know how regression/spawn.sml fairs there.

2. FFI

I am now facing a more complex issue with the foreign function interface. You will find attached a small example (30 lines of C code ; 2 lines of SML code) which reproduces the issue I am facing. In the example, when ffi_func() is called from SML code, it crashes (probably some kind of stack corruption) whereas it runs fine when it is executed from a C main().

This isn't MLton's fault. If you link your test program with -O1 and -lkernel32, you will see the exact same problem with a pure C program. eg:
gcc -Wall -O1 main.c ffi.c -o main.exe -lkernel32
with main.c:

void ffi_func();
int main() { ffi_func(); return 0; }

I have seen this before. With win64 you can't make big allocations on the stack. I had to hack around this for gmp as well. Why gcc gets this wrong, I don't know. If you remove -lkernel32 or use -O2, the problem goes away. Probably your best bet is to not use my_struct as a stack-allocated variable.

I've noticed that mingw (now?) also links to kernel32 on its own. So you could edit the mlton script to not include -lkernel32. I've done a full rebuild and regression test of MLton with -lkernel32 removed, and it worked on my win64 toolchain. If it works for win32 as well, and you can confirm that removing -lkernel32 resolves your FFI problems, I'll probably just remove it in svn too.

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

First, sorry about the multiple spammed replies! Gmail told me it was failing to send and to come back later. When I came back later and sent the definitive copy of my reply, I found that all my earlier clicks had also succeeded. Oops.

On Tue, Aug 12, 2008 at 12:22 AM, Nicolas Bertolotti <Nicolas.Bertolotti@...> wrote:

You can't see it in the sample but the original piece of code on which I observed the crash is from the "zlib" … and I thought I would not need to modify it.

Have you tried removing the -lkernel32? Does that suffice?

You mentioned that you faced a similar issue with GMP. Did you submit a bug report to the GCC team?

Well, the right people to submit the bug report to are the mingw-w64 people, specifically Kai Tietz. I didn't, because at the time I hacked around the GMP problems gcc was so broken that it hardly seemed a useful report. ;-) Now it's a lot better and maybe he should know about this. For gmp, the problem was alloca (which also allocates on the stack). Fortunately, gmp had a compile-time option to change it's use of alloca to malloc. 

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Removing –lkernel32 does not solve the issue.

 

But I could workaround the issue anyway following your other recommendation and replacing the local variable with a pointer and a malloc().

 

I could notice that Kai Tietz had submitted a patch « [patch i386]: x86_64-pc-mingw _alloca and _stkchk may corrupts stack alignment » which seems to be related to the problem you mentionned :

http://gcc.gnu.org/ml/gcc-patches/2008-01/msg01350.html

It seems that this patch should solve the issue with GMP. As I don’t really know how gcc works, I wonder whether it could solve mine as well or if a similar fix should be applied elsewhere.

 

Now I am facing another crash (exit code 128 without any information) in pure SML code so I guess I will come back with additional feedback in a near future.

 

Best regards

 

Nicolas

 

---

From: Wesley W. Terpstra [mailto:wesley@...]
Sent: Tuesday, August 12, 2008 1:27 AM
To: Nicolas Bertolotti; mlton@...
Subject: Re: FW: [MLton] Working: MLton/Win64 toolchain

 

First, sorry about the multiple spammed replies! Gmail told me it was failing to send and to come back later. When I came back later and sent the definitive copy of my reply, I found that all my earlier clicks had also succeeded. Oops.

On Tue, Aug 12, 2008 at 12:22 AM, Nicolas Bertolotti <Nicolas.Bertolotti@...> wrote:

You can't see it in the sample but the original piece of code on which I observed the crash is from the "zlib" … and I thought I would not need to modify it.

Have you tried removing the -lkernel32? Does that suffice?

You mentioned that you faced a similar issue with GMP. Did you submit a bug report to the GCC team?

Well, the right people to submit the bug report to are the mingw-w64 people, specifically Kai Tietz. I didn't, because at the time I hacked around the GMP problems gcc was so broken that it hardly seemed a useful report. ;-) Now it's a lot better and maybe he should know about this. For gmp, the problem was alloca (which also allocates on the stack). Fortunately, gmp had a compile-time option to change it's use of alloca to malloc. 

 

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Wesley W. Terpstra]

2008/8/12 Nicolas Bertolotti <Nicolas.Bertolotti@...>

Now I am facing another crash (exit code 128 without any information) in pure SML code so I guess I will come back with additional feedback in a near future.

Any word on this crash? I need to see it to fix it. :-)

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Nicolas Bertolotti-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

I was actually a problem with GMP that I had forgotten to build with the --enable-alloca=malloc-reentrant option.

 

The remaining issues I am now facing are not related to the Win64 port or even MLton in some cases (scaling issues during refFlatten and deepFlatten ; memory overhead ; call functions in Win64 DLLs)

 

Thanks again for the update. This is already very useful to us.

 

Nicolas

 

---

From: Wesley W. Terpstra [mailto:wesley@...]
Sent: Friday, August 15, 2008 9:43 PM
To: Nicolas Bertolotti; mlton@...
Subject: Re: FW: [MLton] Working: MLton/Win64 toolchain

 

2008/8/12 Nicolas Bertolotti <Nicolas.Bertolotti@...>

Now I am facing another crash (exit code 128 without any information) in pure SML code so I guess I will come back with additional feedback in a near future.

Any word on this crash? I need to see it to fix it. :-)

_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton

[Matthew Fluet-3]

On Fri, 8 Aug 2008, Wesley W. Terpstra wrote:

> I'm happy to announce that MLton now works perfectly on 64-bit windows using
> the native codegen. Please give it a spin and report any problems. It has
> only been tested on Win2003/64 and I would be very interested in hearing
> reports about XP64 and Vista64.
>
> A complete toolchain (no additional software required) can be found at:
> http://www.dvs.tu-darmstadt.de/staff/terpstra/mlton-20080808.win64.zip If
> someone could move this to the Experimental page on the wiki, I would
> appreciate it (I don't have write permissions there). Be warned that the
> toolchain is a 180MB download.

Posted to the Experimental page.


_______________________________________________
MLton mailing list
MLton@...
http://mlton.org/mailman/listinfo/mlton