Makefile Migration from 0.7 to 0.8 questions and suggestions

[MootCycle]

I have recently been working on migrating a qooxdoo 0.7 web application to the 0.8 framework. I've reached the point where I have most of the components transferred using an 0.8 sandbox, and I wanted to convert the main application. I've been reading a lot about the new build system, but I can't seem to get it to do what I want it to do. I expect that I'm missing something simple, and someone who knows the system better than I do can answer these questions very quickly. At the moment, I feel like there's a gap in the documentation specifically in this wiki page:
http://qooxdoo.org/documentation/0.8/migration_makefile
It is linked to from the 0.7 migration guide, but the page is currently empty. I'm specifically interested in how to migrate the old makefile settings APPLICATION_BUILD_PATH, APPLICATION_SOURCE_PATH, APPLICATION_HTML_TO_ROOT_URI, and APPLICATION_PUBLISH_PATH. I think I have found these settings referenced on this page:
http://qooxdoo.org/documentation/0.8/generator_config_ref

I suspect that I am specifying these settings properly (from the old makefile), but in the wrong node or level in the JSON. It isn't clear to me from the examples in the config key reference listing where these settings should sit in the JSON object to be processed. When I run the generate script, it always produces the same output, even when I put intentionally wrong data into the settings. Is it possible to have the generate script give me a warning when it detects a JSON property that is not going to affect the resulting build? If I had that feedback, I would at least know that I had put my settings in the wrong place, rather than a silent error.

I'm sorry if I'm missing something obvious here, but I've spent a lot of time looking over the config key reference and the generator config file wiki pages, and I just don't grok the new system. I can certainly provide specifics about where the files exist if anyone is interested.

Thanks,
-Dan

[MootCycle]

I figured this issue out myself by looking at the example.json, application.json, and the base.json files in the SDK. In case anyone else has the same issue I did, the way I ended up solving this was to define my own job with the appropriate settings in the JSON file and then to call the build source job from my job. There may be another way to do this, but my application is working now. Here's the full config.json file:

{
  "name"    : "client",

  "include" :
  [
    {
      "path" : "../../../../../../qooxdoo-0.8-sdk/tool/data/config/application.json"
    }
  ],

  "let" :
  {
    "APPLICATION"  : "client",
    "QOOXDOO_PATH" : "../../../../../../qooxdoo-0.8-sdk/framework",
    "QOOXDOO_URI"  : "../${QOOXDOO_PATH}",
    "QXTHEME"      : "qx.theme.Modern",
    "API_EXCLUDE"  : ["qx.legacy.*"],
    "LOCALES"      : [ "en" ],
    "ROOT"         : "."
  },
 
  "jobs" :
  {
    "my-source" :
    {
      "compile-source" :
      {
        "file" : "source/script/client.js",
        "root" : "../../../..",
        "locales" : ["en"],
        "gzip" : false
      },
     
      "run" :
        [
          "source"
        ]
    }
  }
}


I hope someone finds this useful. Figuring out how to define and run my own jobs made all the difference in understanding the build system for me.

-Dan

[thron7]

MootCycle wrote:

> I figured this issue out myself by looking at the example.json,
> application.json, and the base.json files in the SDK. In case anyone else
> has the same issue I did, the way I ended up solving this was to define my
> own job with the appropriate settings in the JSON file and then to call the
> build source job from my job. There may be another way to do this, but my
> application is working now. Here's the full config.json file:
>
> {
>   "name"    : "client",
>
>   "include" :
>   [
>     {
>       "path" :
> "../../../../../../qooxdoo-0.8-sdk/tool/data/config/application.json"
>     }
>   ],
>
>   "let" :
>   {
>     "APPLICATION"  : "client",
>     "QOOXDOO_PATH" : "../../../../../../qooxdoo-0.8-sdk/framework",
>     "QOOXDOO_URI"  : "../${QOOXDOO_PATH}",
>     "QXTHEME"      : "qx.theme.Modern",
>     "API_EXCLUDE"  : ["qx.legacy.*"],
>     "LOCALES"      : [ "en" ],
>     "ROOT"         : "."
>   },
>  
>   "jobs" :
>   {
>     "my-source" :
>     {
>       "compile-source" :
>       {
>         "file" : "source/script/client.js",
>         "root" : "../../../..",
>         "locales" : ["en"],
>         "gzip" : false
>       },
>      
>       "run" :
>         [
>           "source"
>         ]
>     }
>   }
> }
>
>
>  

This is stark, Dan! I'm glad it works for you, but I don't recommend
this as a general strategy.

For one thing you defined a "run" job, which is fine. But you have to be
aware that every run job is (recursively) replaced by it's subjobs (the
jobs in the run list). But: Whatever you specify in the initial "run"
job becomes part of the replacing jobs (The "run" job is replaced by as
many copies of itself as there are subjobs, and each copy "extend"s the
corresponding subjob). That means that every job that "my-source" is
eventually replaced by will have a "compile-source" key and will run the
compile task! This is normally not what you want.

In this particular case it didn't matter since the "source" job from
base.json just runs "source-script", which is a "compile-source" job
itself, so you are effectively shaddowing one "compile-source" key with
another, which is fine. But imagine applying the same technique to the
"build" job...

Here's a little quiz for everybody:
How many times would the compiler generate the build/script/<name>.js
file if you applied the same strategy to the "build" job? How could you
overcome this problem?

Cheers,
Thomas

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
qooxdoo-devel mailing list
qooxdoo-devel@...
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

[thron7]

Dan,


> can't seem to get it to do what I want it to do. I expect that I'm missing
> something simple, and someone who knows the system better than I do can
> answer these questions very quickly. At the moment, I feel like there's a
> gap in the documentation specifically in this wiki page:
> http://qooxdoo.org/documentation/0.8/migration_makefile
> It is linked to from the 0.7 migration guide, but the page is currently
> empty. I'm specifically interested in how to migrate the old makefile
> settings APPLICATION_BUILD_PATH, APPLICATION_SOURCE_PATH,
> APPLICATION_HTML_TO_ROOT_URI, and APPLICATION_PUBLISH_PATH.

I have filed bug#1471 to that end.

>  I think I have
> found these settings referenced on this page:
> http://qooxdoo.org/documentation/0.8/generator_config_ref
>
> I suspect that I am specifying these settings properly (from the old
> makefile), but in the wrong node or level in the JSON. It isn't clear to me
> from the examples in the config key reference listing where these settings
> should sit in the JSON object to be processed.

It would help if you could express which information was missing or
incomplete or hard to understand from the docs. E.g. the config overview
has a section that lists all keys with their proper nesting
(http://qooxdoo.org/documentation/0.8/generator_config#listing_of_keys_in_context).
Maybe it is hard to read, but it should give you exactly this information.

>  When I run the generate
> script, it always produces the same output, even when I put intentionally
> wrong data into the settings. Is it possible to have the generate script
> give me a warning when it detects a JSON property that is not going to
> affect the resulting build?

This is a good point. For one thing there is currently AFAIK no
ready-to-use schema system available for Json, so we would have to write
one on our own. For another thing,  I wanted to allow "irregular"
entries so the config handling is more forgiving. Of course, it's also
less informative this way :( .

>  If I had that feedback, I would at least know
> that I had put my settings in the wrong place, rather than a silent error.
>
> I'm sorry if I'm missing something obvious here, but I've spent a lot of
> time looking over the config key reference and the generator config file
> wiki pages, and I just don't grok the new system.

Again, since this is a recurring theme on the list, if you could be more
specific as to where you got stuck, that would be very helpful. I would
be happy to add more and/or better documentation if I knew how to
effectively do that :).

Bear in mind that we implemented the build system in the first go, to
provide the functionality we needed, and in a way we thought would be
conceptually "50% in the right direction" (to quote Richard Gabriel), so
we had something to build upon. But I don't consider it ultimate wisdom
cut into stone, so qualified feedback on both the documentation and the
features is highly appreciated.

>  I can certainly provide
> specifics about where the files exist if anyone is interested.
>  

I don't get this!? Which files do you mean?

Thomas


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
qooxdoo-devel mailing list
qooxdoo-devel@...
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

[MootCycle]

thron7 wrote:

This is stark, Dan! I'm glad it works for you, but I don't recommend
this as a general strategy.

For one thing you defined a "run" job, which is fine. But you have to be
aware that every run job is (recursively) replaced by it's subjobs (the
jobs in the run list). But: Whatever you specify in the initial "run"
job becomes part of the replacing jobs (The "run" job is replaced by as
many copies of itself as there are subjobs, and each copy "extend"s the
corresponding subjob). That means that every job that "my-source" is
eventually replaced by will have a "compile-source" key and will run the
compile task! This is normally not what you want.

In this particular case it didn't matter since the "source" job from
base.json just runs "source-script", which is a "compile-source" job
itself, so you are effectively shaddowing one "compile-source" key with
another, which is fine. But imagine applying the same technique to the
"build" job...

The only reason I ended up writing the config.json file this way was that this was the only way I found that got the "compile-source" key to affect the build process. I tried putting that same "compile-source" key in several different places in my config.json file, and the result was either that it was ignored (thus default values were used for the generate source command) or a fatal error. When I tried to define my own job named "source" and put the key there, I get this: KeyError: u'Job already exists: "source"'. (I was trying to follow the example config file listed here: http://qooxdoo.org/documentation/0.8/generator_config ) So my question boils down to, if I want to provide "compile-source" arguments to the "source" job, how do you recommend I put them in the config.json file I posted earlier?

I've tried putting that key in the top level of the JSON object, in the "jobs" key, and in an individual "job" key. None of those places worked for me unless I defined my own job, which you don't recommend doing. (Also, it doesn't work when I try a build version.)

thron7 wrote:

Here's a little quiz for everybody:
How many times would the compiler generate the build/script/<name>.js
file if you applied the same strategy to the "build" job? How could you
overcome this problem?

Cheers,
Thomas

No idea, which is why I'm posting here. ;)

Thanks for the reply.

-Dan

[thron7]

> The only reason I ended up writing the config.json file this way was that
> this was the only way I found that got the "compile-source" key to affect
> the build process. I tried putting that same "compile-source" key in several
> different places in my config.json file, and the result was either that it
> was ignored (thus default values were used for the generate source command)
> or a fatal error. When I tried to define my own job named "source" and put
> the key there, I get this: KeyError: u'Job already exists: "source"'. (I was
> trying to follow the example config file listed here:
> http://qooxdoo.org/documentation/0.8/generator_config )

This sample doesn't use any external config (there's no top-level
"include" key). But you can easily circumvent the name clash by renaming
your local job to something like "mysource".

>  So my question boils
> down to, if I want to provide "compile-source" arguments to the "source"
> job, how do you recommend I put them in the config.json file I posted
> earlier?
>  

A simple way for doing this would be as follows: You had the "my-source"
job already. Remove the "run" key and add "extend" : ["source-script"]
(provided you are using the trunk version of qooxdoo; otherwise see
further down). This should do the job. For the build version it is more
complex (see also further down).

The point here is that you actually don't want to parameterized the
"source" job with a modified "compile-source" argument, but the
"source-script" job one level further down.

> I've tried putting that key in the top level of the JSON object, in the
> "jobs" key, and in an individual "job" key.

The only valid context for the "compile-source" key is an individual job
(This URL tries to convey this:
http://qooxdoo.org/documentation/0.8/generator_config#listing_of_keys_in_context)

>  None of those places worked for
> me unless I defined my own job, which you don't recommend doing.

Oh yes, I *do* recommend defining own jobs! I just wanted you to be
aware of the peculiarities incurred by using the "run" key within a job.

>  (Also, it
> doesn't work when I try a build version.)
>  

It can't. The "compile-source" key is the key to trigger creation of the
source version. "compile-build" is the key to create the build version,
and it's an independent key. Changes you do to one don't affect the other.

>> Here's a little quiz for everybody:
>> How many times would the compiler generate the build/script/<name>.js
>> file if you applied the same strategy to the "build" job? How could you
>> overcome this problem?
>>
>>    
>
> No idea, which is why I'm posting here. ;)
>  

Too bad nobody anwered to this one (I was silently hoping I could stir a
little playful competition by posting a series of generator quizzes,
thereby fostering more insight in its workings ...:(.

If you created a "my-build" job like "my-build" : { "compile-build" :
..., "run" : ["build"] }, this would have happened: Since the "build"
job from base.json is a run job itself, and is expanded into three
distinct jobs, each of these jobs would have inherited the
"compile-build" key from the "my-build" job. Running this job would have
resulted in the compiler generating the script output three times
(provided there were no errors in between).

You'd overcome this problem by moving the "run" key to a separate local
job with the only key being "run" : ["build-resources", "build-files",
"my-build"], and at the same time defining "my-build" as "my-build" : {
"compile-build" : ...<as before>..., "extend" : ["build-script"] }. This
way, you have a local job with a "run" key that runs the default
build-files, build-resources, but in place of the default build-script
it runs your self-defined my-build. This one contains the compile-dist
key, but makes also use of the settings derived from the original
build-script job. (This only works when the application.json/base.json
have no "export" key restriction on the top-level, which is the case in
the trunk version since recently; you can remove it manually if you have
a 0.8 release version).

HTH,
Thomas




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
qooxdoo-devel mailing list
qooxdoo-devel@...
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel