Getting rid of the "edit-compile-run-cycle" for Kawa

[Ethan Herdrick-2]

Hi -

We really want to improve our dev environment for our Kawa web app.
The main problem is the "edit-compile-run-cycle".  Even though Kawa is
a great language, this style of development makes it feel like we are
just using Java.  It slows us down.  What I want is to change some
function defined in a static module and simply refresh any page in our
app that uses that function and see the result of the change.

In trying to achieve this I've made a simple macro "require/smart"
that will expand to (require <foo>) if the system is in production
mode, and expand to (load "\some-path\foo.scm") if in development
mode.  Of course, our existing code behaves somewhat differently when
loaded instead of required.  I've been working through this but I'm
not sure what the ultimate result will be.  I might be overlooking
some fatal flaw in this plan.  This technique does work fine in a very
simple proof of concept with a very simple tree of dependencies made
up of three simple files of definitions.

In an old post to this list (found here:
http://www.sourceware.org/ml/kawa/2006-q4/msg00002.html )  Per wrote:

"...you can:  (require "file-name")  and it automatically compile sit,
as a module, and recompiles if it has changed.  It updates the global
environment. (However, there is no support for updating existing
values -such as if an existing value references an updated function -
but that could be added.)"

That would be fantastic.  What would need to be done to add that?
Could it understand dependencies, i.e. recompile a class if a class it
depends on has been recompiled?  That kind of dynamic dev environment
- combined with Kawa's impressive performance as a compiled language -
would really be amazing.

Thanks in advance for your thoughts!

-Ethan

--
Ethan Herdrick
www.biographicon.com - Everyone's biography.

[Helmut Eller-2]

* Ethan Herdrick [2008-05-03 14:21+0200] writes:

[...]
> In trying to achieve this I've made a simple macro "require/smart"
> that will expand to (require <foo>) if the system is in production
> mode, and expand to (load "\some-path\foo.scm") if in development
> mode.  Of course, our existing code behaves somewhat differently when
> loaded instead of required.  I've been working through this but I'm
> not sure what the ultimate result will be.  I might be overlooking
> some fatal flaw in this plan.  This technique does work fine in a very
> simple proof of concept with a very simple tree of dependencies made
> up of three simple files of definitions.

I also think that Kawa needs improvements in this area.  I'd love to see
support for R6RS/ERR5RS style modules, which, I think, would solve some
of the dependency tracking problems.

> In an old post to this list (found here:
> http://www.sourceware.org/ml/kawa/2006-q4/msg00002.html )  Per wrote:
>
> "...you can:  (require "file-name")  and it automatically compile sit,
> as a module, and recompiles if it has changed.  It updates the global
> environment. (However, there is no support for updating existing
> values -such as if an existing value references an updated function -
> but that could be added.)"
>
> That would be fantastic.  What would need to be done to add that?
> Could it understand dependencies, i.e. recompile a class if a class it
> depends on has been recompiled?  That kind of dynamic dev environment
> - combined with Kawa's impressive performance as a compiled language -
> would really be amazing.
>
> Thanks in advance for your thoughts!

I think recompiling alone isn't enough.  The existing instances of the
old version should be updated too.  Some JVMs support class
redefinition.  That could be used, but it's probably the sledgehammer
solution.

For my needs it would be sufficient if all top-level-functions in a
module --also those which aren't exported-- were indirectly called
through a location.  [A bit like in Common Lisp where
top-level-functions are indirectly called through the symbol-function
slot.]  When it's time to upgrade a module the existing locations must
be updated with the new values, so that calls are redirected to the new
version.

This should also somehow work for multiple threads.  After updating a
module I usually want that all threads use the new version.  

Just my 2 Cents.

Helmut.

[Per Bothner]

Ethan Herdrick wrote:
> Hi -
>
> We really want to improve our dev environment for our Kawa web app.
> The main problem is the "edit-compile-run-cycle".  Even though Kawa is
> a great language, this style of development makes it feel like we are
> just using Java.  It slows us down.  What I want is to change some
> function defined in a static module and simply refresh any page in our
> app that uses that function and see the result of the change.

Have you tried this:
http://www.gnu.org/software/kawa/server/auto-servlet.html

[One of the problems I need to fix with Kawa is that finding
documentation isn't always easy.  This particular file is an example
of that ...]

I haven't tested this recently, but hopefully it still works.
Please let me know.  I also have some further serlvet-related
patches from various past experiments - these need to be re-tested
and integrated.

> In trying to achieve this I've made a simple macro "require/smart"
> that will expand to (require <foo>) if the system is in production
> mode, and expand to (load "\some-path\foo.scm") if in development
> mode.  Of course, our existing code behaves somewhat differently when
> loaded instead of required.  I've been working through this but I'm
> not sure what the ultimate result will be.  I might be overlooking
> some fatal flaw in this plan.  This technique does work fine in a very
> simple proof of concept with a very simple tree of dependencies made
> up of three simple files of definitions.

I always discourage use of load.

> In an old post to this list (found here:
> http://www.sourceware.org/ml/kawa/2006-q4/msg00002.html )  Per wrote:
>
> "...you can:  (require "file-name")  and it automatically compile sit,
> as a module, and recompiles if it has changed.  It updates the global
> environment. (However, there is no support for updating existing
> values -such as if an existing value references an updated function -
> but that could be added.)"
>
> That would be fantastic.  What would need to be done to add that?

It should work now - though I just found a bug ...

> Could it understand dependencies, i.e. recompile a class if a class it
> depends on has been recompiled?

It does that, if you replace "class" by "module": A module A requires B,
and B requires C, the way it is supposed to happen is this:
If a module A requires A, it will make a dependent of B and C.
Next time the require is "executed", then the dependence of B and C
is checked.  If C is modified, this will before B to be recompiled,
which will force A to be recompiled.

Or something like that - I haven't looked at it in a while.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Per Bothner]

Helmut Eller wrote:
> I also think that Kawa needs improvements in this area.  I'd love to see
> support for R6RS/ERR5RS style modules, which, I think, would solve some
> of the dependency tracking problems.

Kawa does support the concept but not the syntax of R6RS/ERR5RS style
modules.  Kawa doesn't support selective import (only importing some
of the exported names) or renaming on import.  I don't think those
would be difficult to add.

> I think recompiling alone isn't enough.  The existing instances of the
> old version should be updated too.

That's harder.  A partial solution would be if the body of a function
changes, to modify the function in place so the old function object
now has a new body.

> Some JVMs support class
> redefinition.  That could be used, but it's probably the sledgehammer
> solution.

It would be nice to look into the options

> For my needs it would be sufficient if all top-level-functions in a
> module --also those which aren't exported-- were indirectly called
> through a location.  [A bit like in Common Lisp where
> top-level-functions are indirectly called through the symbol-function
> slot.]  When it's time to upgrade a module the existing locations must
> be updated with the new values, so that calls are redirected to the new
> version.

All the top-level functions are represented by a ModuleMethod object,
which provides the needed re-direction.  We do need to limit the
inlining so calls work through the ModuleMethod, rather than going
all the way down to the static method.

> This should also somehow work for multiple threads.  After updating a
> module I usually want that all threads use the new version.

That would heppen automatically if they use the ModuleMethod for calls.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Ethan Herdrick-3]

Yes, we've been using KawaPageServlet with automatic servlet mapping,
and it works great, as documented.  But it isn't giving us automatic
recompile of static modules.  Definitions made inside the servlet
itself are immediately reflected, however.

> It does that, if you replace "class" by "module": A module A requires B,
> and B requires C...

I'm not seeing that behavior.  For one thing, I'm getting the (require
"foo.scm") usage to work from within a servlet.  But if that file,
"foo.scm" contains a (require "bar.scm"), I can't get that second
require to work.  What should "bar.scm" be?  An absolute path, like
"/usr/tomcat/webapps/someapp/ROOT/WEB-INF/classes/bar.scm"?  Or a path
relative to the calling file's path?  I've tried absolute and relative
paths and only the absolute paths are working for calls to require
from within servlets, but as I mention above, nothing is working for
me when calling from a static module file.

I should mention that I'm using Tomcat on Windows XP.  I've tried this
with both Kawa 1.9.1 and the latest build from source.

There are other problems but I'll wait until I get this path thing sorted out.
Thank you!
-Ethan

On Sat, May 3, 2008 at 8:49 AM, Per Bothner <per@...> wrote:

> Ethan Herdrick wrote:
>
> > Hi -
> >
> > We really want to improve our dev environment for our Kawa web app.
> > The main problem is the "edit-compile-run-cycle".  Even though Kawa is
> > a great language, this style of development makes it feel like we are
> > just using Java.  It slows us down.  What I want is to change some
> > function defined in a static module and simply refresh any page in our
> > app that uses that function and see the result of the change.
> >
>
>  Have you tried this:
>  http://www.gnu.org/software/kawa/server/auto-servlet.html
>
>  [One of the problems I need to fix with Kawa is that finding
>  documentation isn't always easy.  This particular file is an example
>  of that ...]
>
>  I haven't tested this recently, but hopefully it still works.
>  Please let me know.  I also have some further serlvet-related
>  patches from various past experiments - these need to be re-tested
>  and integrated.
>
>
>
> > In trying to achieve this I've made a simple macro "require/smart"
> > that will expand to (require <foo>) if the system is in production
> > mode, and expand to (load "\some-path\foo.scm") if in development
> > mode.  Of course, our existing code behaves somewhat differently when
> > loaded instead of required.  I've been working through this but I'm
> > not sure what the ultimate result will be.  I might be overlooking
> > some fatal flaw in this plan.  This technique does work fine in a very
> > simple proof of concept with a very simple tree of dependencies made
> > up of three simple files of definitions.
> >
>
>  I always discourage use of load.
>
>
>
> > In an old post to this list (found here:
> > http://www.sourceware.org/ml/kawa/2006-q4/msg00002.html )  Per wrote:
> >
> > "...you can:  (require "file-name")  and it automatically compile sit,
> > as a module, and recompiles if it has changed.  It updates the global
> > environment. (However, there is no support for updating existing
> > values -such as if an existing value references an updated function -
> > but that could be added.)"
> >
> > That would be fantastic.  What would need to be done to add that?
> >
>
>  It should work now - though I just found a bug ...
>
>
>
> > Could it understand dependencies, i.e. recompile a class if a class it
> > depends on has been recompiled?
> >
>
>  It does that, if you replace "class" by "module": A module A requires B,
>  and B requires C, the way it is supposed to happen is this:
>  If a module A requires A, it will make a dependent of B and C.
>  Next time the require is "executed", then the dependence of B and C
>  is checked.  If C is modified, this will before B to be recompiled,
>  which will force A to be recompiled.
>
>  Or something like that - I haven't looked at it in a while.
>  --
>         --Per Bothner
>  per@...   http://per.bothner.com/
>

--
Ethan Herdrick
www.biographicon.com - Everyone's biography.

[Per Bothner]

Ethan Herdrick wrote:

> > It does that, if you replace "class" by "module": A module A requires B,
>> and B requires C...
>
> I'm not seeing that behavior.  For one thing, I'm getting the (require
> "foo.scm") usage to work from within a servlet.  But if that file,
> "foo.scm" contains a (require "bar.scm"), I can't get that second
> require to work.  What should "bar.scm" be?  An absolute path, like
> "/usr/tomcat/webapps/someapp/ROOT/WEB-INF/classes/bar.scm"?  Or a path
> relative to the calling file's path?  I've tried absolute and relative
> paths and only the absolute paths are working for calls to require
> from within servlets, but as I mention above, nothing is working for
> me when calling from a static module file.

I'm looking into it.  I thought this was supposed to work, but
perhaps I was just optimistic or somehow it broke.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Per Bothner]

The last few days I've done some work on modules and dependencies.
Things seems to work better now - please try again with the current
SVN source.

Ethan Herdrick wrote:
> I'm not seeing that behavior.  For one thing, I'm getting the (require
> "foo.scm") usage to work from within a servlet.  But if that file,
> "foo.scm" contains a (require "bar.scm"), I can't get that second
> require to work.

It seems to work now, at least for me.

> What should "bar.scm" be?  An absolute path, like
> "/usr/tomcat/webapps/someapp/ROOT/WEB-INF/classes/bar.scm"?  Or a path
> relative to the calling file's path?

I guess the former will work, but the latter is of course recommended.

> I've tried absolute and relative
> paths and only the absolute paths are working for calls to require
> from within servlets, but as I mention above, nothing is working for
> me when calling from a static module file.

I haven't tried a "static module file", but this works:

adder.scm:

(require 'http)
(require "result.scm")
(define (num-parameter name default)
   (string->number (request-parameter name default)))

(define sum1 (num-parameter 'sum1 0))
(define sum2 (num-parameter 'sum2 1))

(html:html
  (html:head (html:title "Accumulating adder"))

  (html:body
   (html:form method:'POST
    (html:table
     (html:tr

      (html:td (result-so-far))
      (html:td
       (html:input
        name: 'sum1
        value: (+ sum1 sum2))))
     (html:tr
      (html:td "Add to it:")
      (html:td
       (html:input name: 'sum2 value: sum2))))
    (html:input type:"submit" value:"Submit"))))

result.scm:

(define (result-so-far)
   "The result so far is: ")

If I modify the "The result so far is: " then that
get reflected when I reload adder.scm.

It also works if I rename "adder.scm" to plain "adder",
as long as the file starts with:

;; This is -*- scheme -*-

As you suggest, there may also be some Windows-specific
problems - I've only tested this on GNU/Linux.  But first
see if the changes I made have helped.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Per Bothner]

Ethan Herdrick wrote:
> That would be fantastic.  What would need to be done to add that?
> Could it understand dependencies, i.e. recompile a class if a class it
> depends on has been recompiled?  That kind of dynamic dev environment
> - combined with Kawa's impressive performance as a compiled language -
> would really be amazing.

That seems to work now, if you:

- Use the automatic servlet mapping as described here:
http://www.gnu.org/software/kawa/server/auto-servlet.html

- Use require with a source file name (normally relative)
to "import" any dependencies.

Then a source file is automatically recompiled when it or
one of its dependencies changes.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Ethan Herdrick-2]

OK requiring 'http works for me, too.

However, the original bug (builtin functions that produce HTML don't
work with higher-order functions) is still there for me.  For example:

;; -*- scheme -*-
(require 'http)
(apply html:b '("bold this " "and this ")) ; works
(list (html:b "also this ") (html:b "this too ")) ; works
(map html:b '("bold 1" "bold2")) ; fails.  gives:
jndi:/mon.dev.biographicon.com/temp2:7:1: called value is not a
procedure
#\newline


If I comment out that last line the servlet works as expected.  So
apply is working with html:b but map is not.  This is with the latest
snapshot.



On Tue, May 20, 2008 at 10:53 AM, Per Bothner <per@...> wrote:

> It's seems (require 'http) is required.  This works:
>
> ;; -*- scheme -*-
> (require 'http)
> "hello world"
> #\newline
>
> I haven't figured out why yet.
> --
>        --Per Bothner
> per@...   http://per.bothner.com/
>

--
Ethan Herdrick
www.biographicon.com - Everyone's biography.

[Per Bothner]

Ethan Herdrick wrote:
> OK requiring 'http works for me, too.

I just check in a fix for this.

>
> However, the original bug (builtin functions that produce HTML don't
> work with higher-order functions) is still there for me.  For example:
>
> ;; -*- scheme -*-
> (require 'http)
> (apply html:b '("bold this " "and this ")) ; works
> (list (html:b "also this ") (html:b "this too ")) ; works
> (map html:b '("bold 1" "bold2")) ; fails.  gives:
> jndi:/mon.dev.biographicon.com/temp2:7:1: called value is not a
> procedure
> #\newline
>
>
> If I comment out that last line the servlet works as expected.  So
> apply is working with html:b but map is not.  This is with the latest
> snapshot.

This is a consequence of the "generalized apply" - we now allow
application of types, vectors, and more.  But the code-path
taken when compiling map doesn't handle this.  Disabling the
error message (and just return the exp unmodified) causes a
failure later.  I'll look into it.
--
        --Per Bothner
per@...   http://per.bothner.com/

[Per Bothner]

> ;; -*- scheme -*-
> (require 'http)
> (apply html:b '("bold this " "and this ")) ; works
> (list (html:b "also this ") (html:b "this too ")) ; works
> (map html:b '("bold 1" "bold2")) ; fails.  gives:
> jndi:/mon.dev.biographicon.com/temp2:7:1: called value is not a
> procedure
> #\newline
>
>
> If I comment out that last line the servlet works as expected.  So
> apply is working with html:b but map is not.  This is with the latest
> snapshot.

I checked in a fix for this.
--
        --Per Bothner
per@...   http://per.bothner.com/