Broken Release Process [was: Re: Displaying an animation / "movie"]

[Søren Hauberg]

Moving this to Octave-Forge mailing list from help@octave...

Quoting Thomas Weber <thomas.weber.mail@...>:
> I don't intend to add any more octave-forge packages to Debian until
> octave-forge's release process is fixed (and yes, it currently is
> broken).
>
> By "fixed" I mean that:
> 1) Packages that had no changes are not released with a new version
> number.

Yes, this part of the process is just very broken. It seems that the  
release process currently depends on the machine used to make the  
release. This is one of the reasons why being a release manager sucks...

> 2) The total number of packages is reduced.

I agree that some packages should probably be merged, but I don't  
think we should hope for fewer packages. Personally, I'm hoping we'll  
get thousands of new contributors, that will create a bunch of high  
quality packages for us all (this might be slightly optimistic, though  
:-) ).

> 1) Updating a package in Debian takes a minimal amount of time
> regardless of how little changed. With 50 octave-forge packages, I spend
> more than a whole day updating these packages. It already costs me a
> whole day and we don't even have all packages in Debian.
> That's just insane, given that a lot of these packages haven't changed
> for over a year (and that's a conservative estimate, see below).

I feel your pain, Brother :-)
Just out of curiosity: how automatic is your build process? Does it  
require much manual labor?

> And no, the bundle is not a solution to that; it's just covering the
> problem, not solving it.

I agree. The bundle only reduces the amount of downloads needed, which  
isn't really related to the real problem.

But what is the solution? In my mind, package maintainers should also  
be release managers for their own packages. But how can we make this  
possible? My proposal is the following work flow:

   1) Maintainer X decides to make a new release of SomePackage.
   2) Maintainer X creates the package file, performs test, and so on.
   3) Maintainer X creates the web pages related to the package (online docs).
   4) Maintainer X uploads the package and the web pages to an FTP server, and
      notifies the Octave-Forge maintainers (me, David, and possibly others)
      that a new version of SomePackage is ready.
   5) The Octave-Forge maintainers uploads the package and the web pages to
      Octave-Forge.

The benefits of this work flow is that the work required by the  
Octave-Forge maintainers is minimal, and package maintainers get more  
control of when new releases are made.

However, there are some problems:
   1) We don't have an FTP server where maintainers can upload their  
stuff. I don't know where we can get one.
   2) We don't have the tools to generate web pages from a package.  
I'm currently working on this. I have something that mostly works  
right now, but I'm quite busy this week, so I won't have time to make  
it public until sometime next week (let me know if you want to see  
half-finished code).
   3) The Octave-Forge maintainers still has to do some manual work  
when a new package is released. This is a potential bottle neck, but I  
don't see a good solution at the moment.
   4) I've removed the bundle from the above work flow. We can  
probably create scripts that updates this bundle, but for now I've  
chosen to ignore it, as I don't consider it very important.

Any thoughts?
Søren


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Thomas Weber-8]

Am Dienstag, den 16.09.2008, 10:54 +0200 schrieb soren@...:

> Moving this to Octave-Forge mailing list from help@octave...
>
> Quoting Thomas Weber <thomas.weber.mail@...>:
> > I don't intend to add any more octave-forge packages to Debian until
> > octave-forge's release process is fixed (and yes, it currently is
> > broken).
> >
> > By "fixed" I mean that:
> > 1) Packages that had no changes are not released with a new version
> > number.
>
> Yes, this part of the process is just very broken. It seems that the  
> release process currently depends on the machine used to make the  
> release. This is one of the reasons why being a release manager sucks...

It's actually too early on announcing this, but I'm working on some
scripts to get rid of the role of octave-forge's release manager
completely, at least in its current form.

That means that
a) The doxygen documentation is updated automatically when a new Octave
version hits the GNU mirrors.
b) Individual packages are built when their maintainer decides to do so
(by committing a release message into SVN).
c) A bundle is created every X months by taring the then current
packages.
d) Everything else that can be reasonably automated ...

I've got a) mostly working (although not with the current octave-forge
HTML layout).
b) is in a prototype form, and will probably need a hgsvn mirror, but
that is a one-time conversion (and then tracking the SVN system further,
but hgsvn can do this easily).

The main obstacle here is SF: in order to do this stuff, one needs a
host with Doxygen and Octave installed. I can build this on tw-math.de,
but getting it to the SF servers is something different.

I'm however not longer willing to tolerate this frustrating process for
everyone involved just because we host stuff on SF.

> > 1) Updating a package in Debian takes a minimal amount of time
> > regardless of how little changed. With 50 octave-forge packages, I spend
> > more than a whole day updating these packages. It already costs me a
> > whole day and we don't even have all packages in Debian.
> > That's just insane, given that a lot of these packages haven't changed
> > for over a year (and that's a conservative estimate, see below).
>
> I feel your pain, Brother :-)
> Just out of curiosity: how automatic is your build process? Does it  
> require much manual labor?

Depends. If a package builds without problems and has no bugs in Debian
reported, it's pretty short. The longest manual time is then signing the
package (it usually takes me 2-3 attempts to type my gpg passphrase
correctly :)).

If there are bugs involved, it becomes more work: checking the Debian
BTS, testing whether the bug is fixed, ...

As soon as manual work is involved, it's time consuming.

> But what is the solution? In my mind, package maintainers should also  
> be release managers for their own packages. But how can we make this  
> possible? My proposal is the following work flow:
>
>    1) Maintainer X decides to make a new release of SomePackage.
>    2) Maintainer X creates the package file, performs test, and so on.
>    3) Maintainer X creates the web pages related to the package (online docs).
>    4) Maintainer X uploads the package and the web pages to an FTP server, and
>       notifies the Octave-Forge maintainers (me, David, and possibly others)
>       that a new version of SomePackage is ready.
>    5) The Octave-Forge maintainers uploads the package and the web pages to
>       Octave-Forge.

Eh, I'm thinking more of:
1) Maintainer commits a "Release PACKAGE" message, with PACKAGE the
package's name. A commit hook in the version control system checks each
commit message for such a phrase and triggers a package build *on the
server* and an update of the web pages. The readily built package is
copied into a publically available FTP/HTTP directory.

        Thomas


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Søren Hauberg]

Quoting Thomas Weber <thomas.weber.mail@...>:
> Eh, I'm thinking more of:
> 1) Maintainer commits a "Release PACKAGE" message, with PACKAGE the
> package's name. A commit hook in the version control system checks each
> commit message for such a phrase and triggers a package build *on the
> server* and an update of the web pages. The readily built package is
> copied into a publically available FTP/HTTP directory.

I'm guessing this requires a bunch of software to be available on the  
server, but lets ignore that for now. What would happen if things  
don't build? I experience this quite a lot. While I would love a more  
automated approach, it somewhat scares me, as my experience is that  
things are quite unstable (this is of course related to the fact that  
our scripts for managing the release is a mess).

Søren


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev

[Thomas Weber-8]

Am Dienstag, den 16.09.2008, 11:32 +0200 schrieb soren@...:

> Quoting Thomas Weber <thomas.weber.mail@...>:
> > Eh, I'm thinking more of:
> > 1) Maintainer commits a "Release PACKAGE" message, with PACKAGE the
> > package's name. A commit hook in the version control system checks each
> > commit message for such a phrase and triggers a package build *on the
> > server* and an update of the web pages. The readily built package is
> > copied into a publically available FTP/HTTP directory.
>
> I'm guessing this requires a bunch of software to be available on the  
> server, but lets ignore that for now.

No need to ignore it, I have that software installed on tw-math.de

> What would happen if things don't build? I experience this quite a lot.

That's bad; however, as we build only individual packages, only that
package build will fail. This must then be investigated, but that's the
job of the person who triggered the build  (and mayme me, if it's a
problem on the server).


> While I would love a more  
> automated approach, it somewhat scares me, as my experience is that  
> things are quite unstable (this is of course related to the fact that  
> our scripts for managing the release is a mess).

These are bugs that must get fixed; yeah, I know, it sounds stupid, but
that's what it boils down to. If it doesn't build on the server, it
doesn't build on your home machine, so the only question is where the
bug is fixed.

        Thomas


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Octave-dev mailing list
Octave-dev@...
https://lists.sourceforge.net/lists/listinfo/octave-dev