3.1 status report

[John W. Eaton]

Here is an update of the goals for 3.1 that I posted here:

  http://www.cae.wisc.edu/pipermail/octave-maintainers/2007-December/005354.html

I've tagged completed items with an "X" and items for which some work
has been done with a "+".  Please let me know if I've missed anything.

   1. Objects:
        X Merge object-branch
        - Implementation of superiorto/inferiorto
        - Operator overloading vs. constant folding

   +. Handle block comments (not finished: block comments inside []
      or {}, and also in the group of comments following a
      continuation character, etc.  See the FIXME comments in lex.l).

   X. Eliminate __gnuplot_X__ functions from Octave

   4. Move code to external packages
        - control
        - finance
        - optimization?
        - signal?
        - statistics?
        - quaternions

   +. Move imread/imwrite functions from Octave Forge to Octave.

   +. Improve traceback error messages.  The messages should list the
      exact line where an error occurred, then only print function
      traceback info and not information about which
      if/for/while/etc. statement includes the error.  That extra
      information just seems to confuse users, and the current
      messages don't always clearly describe the actual location of
      the error.

   X. Ensure that all operations which work on dimensions alone
      (squeeze, numel, triu, etc.) work for all objects and preserve
      type.  Should these functions all be built-in?  Possibly they
      should all be provided by the octave_value class interface.

   X. Make mapper functions work like other built-in functions?

   X. Mapper functions like real, imag, and mod should preserve type
      (are there others?)

  10. Improve compatibility of fsolve
        - input/output args should be compatible
        - use optimset for options

  11. Graphics:
        + Refactor base_properties
        + Specific types for properties with improved property value
          checking
        - Implement the addprops function allow additional properties
          to objects
        - add the hggroup object that has no fixed properties for use
          by barseries, etc.
        - Add callback DeleteFcn/CreateFcn to objects
        - Allow listener functions to be added to objects
        + Clean separation of backend from property database
        + Implement experimental backend based on OpenGL and GUI
          toolkit

Is anyone interested in helping to move the control, finance, and
quaternion functions into separate packages?

Should we move the statistics, signal, and optimization functions to
separate packages?

I'm still hopeful that we can have a release within a month or two.
Although it may contain the OpenGL graphics code, I'm not expecting
that we will make that the default graphics backend.

Would someone like to work on an imwrite function?  I don't think the
current code in Octave Forge is what we want.  Instead, I think that
we should just use GraphicsMagick++ to write the file.  It should be
relatively easy to do that, but to make imwrite compatible requires a
bit of work to properly decode the type and size of the argument and
decide how to store the data in a Magick::Image object so that it can
be written to a file.

Another small but fairly important project is improving the
compatibility of fsolve.  Is anyone interested in working on that?

I hope to have some time to work on items 1, 2, and 6, but if someone
else would like to work on them, please let me know so we don't
duplicate efforts.

I'd also like to start making snapshots again starting this week.  Is
there any objection to doing that?  Unless someone has a better idea,
I was thinking of starting the series of snapshots with version 3.1.50
and making the next stable release 3.2.0.

Comments?

Thanks,

jwe

[Michael Goffioul-2]

On Wed, Jul 16, 2008 at 8:34 PM, John W. Eaton <jwe@...> wrote:

>  11. Graphics:
>        + Refactor base_properties
>        + Specific types for properties with improved property value
>          checking
>        - Implement the addprops function allow additional properties
>          to objects
>        - add the hggroup object that has no fixed properties for use
>          by barseries, etc.
>        - Add callback DeleteFcn/CreateFcn to objects
>        - Allow listener functions to be added to objects
>        + Clean separation of backend from property database
>        + Implement experimental backend based on OpenGL and GUI
>          toolkit

All points above should be marked with a +: there is already
addproperty, addlistener, CreateFcn and DeleteFcn. However,
addproperty and addlistener are not compatible with Matlab
(these have been put as part of the API only in recent versions
of Matlab).

Michael.

[Søren Hauberg]

ons, 16 07 2008 kl. 14:34 -0400, skrev John W. Eaton:
> I'd also like to start making snapshots again starting this week.  Is
> there any objection to doing that?  Unless someone has a better idea,
> I was thinking of starting the series of snapshots with version 3.1.50
> and making the next stable release 3.2.0.
>
> Comments?

I'm a bit curious, but why should the first 3.1.x release be called
3.1.50? Are you so fond of the 2.1.50 release (the one that'll never go
away, it seems...) that you want a similar release in the 3.1.x
series :-)

Anyway, seeing your list really shows the improvements in the 3.1.x
series. Makes me feel kinda guilty that I haven't been able to
contribute -- but great work guys :-)

Søren

[John W. Eaton]

On 16-Jul-2008, Michael Goffioul wrote:

| On Wed, Jul 16, 2008 at 8:34 PM, John W. Eaton <jwe@...> wrote:
| >  11. Graphics:
| >        + Refactor base_properties
| >        + Specific types for properties with improved property value
| >          checking
| >        - Implement the addprops function allow additional properties
| >          to objects
| >        - add the hggroup object that has no fixed properties for use
| >          by barseries, etc.
| >        - Add callback DeleteFcn/CreateFcn to objects
| >        - Allow listener functions to be added to objects
| >        + Clean separation of backend from property database
| >        + Implement experimental backend based on OpenGL and GUI
| >          toolkit
|
| All points above should be marked with a +: there is already

OK, I updated my list.

jwe

[dbateman]

John W. Eaton wrote:

Is anyone interested in helping to move the control, finance, and
quaternion functions into separate packages?

Sure I can take that one on. Is the goal to transfer these to octave-forge, or make them packages that just happen to be part of Octave itself? It would be easy to just transfer them to Octave as the machinery for packages is already in place there. Though if we do this could we have a 3.0.2 release and then an octave-forge release and then we can convert octave-forge for a 3.1 release including

* remove files like num2hex, imread, etc
* 3.1 compatible mapper functions for user types
*etc

and at the same time move the packages fro Octave

Should we move the statistics, signal, and optimization functions to
separate packages?

I thought the consenus was if we do, we should be careful of what functions we move. Maybe we need to discuss all of the candidate functions to remove from Octave and the reasons to keep each of them in Octave itself.

I'd also like to start making snapshots again starting this week.  Is
there any objection to doing that?  Unless someone has a better idea,
I was thinking of starting the series of snapshots with version 3.1.50
and making the next stable release 3.2.0.

Why 3.1.50 and not 3.1.1? not that it matters.

In any case the list looks a bit like a new feature list of the 3.2 NEWS file. In that case maybe the compound operators and single precision type should be mentioned as major new features. As for the single precision type there are 3 things still to do

+ Treat conversion of NA values from single/double and R compatibility
- Convert methods like extract_row from derived Array classes to Array<T> itself
- single precision versions of fsolve, dasrt, etc

I'll definitely address these before 3.2.0, but for now they can wait.

D.

[John W. Eaton]

On 16-Jul-2008, Søren Hauberg wrote:

| I'm a bit curious, but why should the first 3.1.x release be called
| 3.1.50? Are you so fond of the 2.1.50 release (the one that'll never go
| away, it seems...) that you want a similar release in the 3.1.x
| series :-)

I want to make a series of snapshots before the next stable version is
released.  Originally, I was planning to call the next major stable
release 3.1.0, but if we do that, I'm not sure what version to use for
the snapshots.  So I thought we could use 3.1.X for the snapshots and
3.2.0 for the next stable release.  I chose X == 50 because I think we
are at least 50 percent of the way to 3.2.0.  :-)  But we can start
with X at 0 if people think that makes more sense.

jwe

[John W. Eaton]

On 16-Jul-2008, dbateman wrote:

| John W. Eaton wrote:
| >
| > Is anyone interested in helping to move the control, finance, and
| > quaternion functions into separate packages?
| >
|
| Sure I can take that one on. Is the goal to transfer these to octave-forge,
| or make them packages that just happen to be part of Octave itself? It would
| be easy to just transfer them to Octave as the machinery for packages is
| already in place there.

I think I'd rather have them as separate packages.  Moving them to
Octave Forge would be fine with me, and it would be great if you could
do that.

| Though if we do this could we have a 3.0.2 release
| and then an octave-forge release

OK.

| and then we can convert octave-forge for a
| 3.1 release including
|
| * remove files like num2hex, imread, etc
| * 3.1 compatible mapper functions for user types
| *etc
|
| and at the same time move the packages fro Octave

Yes, that would be good.

| > Should we move the statistics, signal, and optimization functions to
| > separate packages?
| >
|
| I thought the consenus was if we do, we should be careful of what functions
| we move. Maybe we need to discuss all of the candidate functions to remove
| from Octave and the reasons to keep each of them in Octave itself.

I think we can delay this until later (maybe never).

| In any case the list looks a bit like a new feature list of the 3.2 NEWS
| file. In that case maybe the compound operators and single precision type
| should be mentioned as major new features. As for the single precision type
| there are 3 things still to do

Yes, we also need to update the NEWS file.  In the future, I think we
should probably try to be more disciplined about adding NEWS entries
as changes are made (I'm definitely guilty of not adding entries for
the changes I make).  Otherwise, creating the NEWS file for a release
is more difficult.

| + Treat conversion of NA values from single/double and R compatibility
| - Convert methods like extract_row from derived Array classes to Array<T>
| itself
| - single precision versions of fsolve, dasrt, etc
|
| I'll definitely address these before 3.2.0, but for now they can wait.

OK.

Thanks,

jwe

[Jaroslav Hajek-2]

>  10. Improve compatibility of fsolve
>        - input/output args should be compatible
>        - use optimset for options
>

I started to work on this, as well as adding/porting other functions
(fzero, lsqnonlin etc).
But I caught some rumors about 4. (moving optimization to OctaveFOrge
package) and I stopped until this is resolved. Personally, I'd vote
for having the basic nonlinear optimization/least squares/root finding
directly in Octave.
To be more constructive, I propose:

1. Update optimset to be reasonably Matlab-compatible (differences are
unavoidable due to different algorithms employed)

2. Port the 1-dimensional funcs fzero and fminbnd from octave-forge,
possibly update.

3a. Include the rest of MINPACK (currently used for fsolve) to
implement lsqnonlin. Also modify the MINPACK drivers to reverse
communication so as to make the functions reentrant.
OR
3b. Implement fsolve and lsqnonlin drivers as m-files, possibly
wrapping the MINPACK step selection subprograms and reusing them.
More work, but probably a more maintainable and extendable outcome.

4. improve lsqnonneg using the QR updating funcs (they can also be
used for the Powell hybrid method in fsolve).

I'd tend to leave the general multidimensional optimization codes like
fmins, bfgs etc. in Octave Forge. Many algorithms exist in this area
with no clear winner for Octave, IMHO.

all of the above seems futile if the long-term goal is to move the
optimization funcs away from core Octave, so I'd appreciate if more
people vote for this plan before I resume the work.

cheers


--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

[John W. Eaton]

On 16-Jul-2008, Jaroslav Hajek wrote:

| >  10. Improve compatibility of fsolve
| >        - input/output args should be compatible
| >        - use optimset for options
| >
|
| I started to work on this, as well as adding/porting other functions
| (fzero, lsqnonlin etc).
| But I caught some rumors about 4. (moving optimization to OctaveFOrge
| package) and I stopped until this is resolved. Personally, I'd vote
| for having the basic nonlinear optimization/least squares/root finding
| directly in Octave.

I agree that Octave should have some general-purpose optimization
and root finding functions included by default.

| To be more constructive, I propose:
|
| 1. Update optimset to be reasonably Matlab-compatible (differences are
| unavoidable due to different algorithms employed)
|
| 2. Port the 1-dimensional funcs fzero and fminbnd from octave-forge,
| possibly update.
|
| 3a. Include the rest of MINPACK (currently used for fsolve) to
| implement lsqnonlin. Also modify the MINPACK drivers to reverse
| communication so as to make the functions reentrant.
| OR
| 3b. Implement fsolve and lsqnonlin drivers as m-files, possibly
| wrapping the MINPACK step selection subprograms and reusing them.
| More work, but probably a more maintainable and extendable outcome.
|
| 4. improve lsqnonneg using the QR updating funcs (they can also be
| used for the Powell hybrid method in fsolve).

These all seem OK to me.  If you are writing the code, then I think it
is up to you to choose between 3a and 3b.  Does GSL have a C
translation of the Minpack algorithms?  If so, maybe we could use
those.  If we choose that, then item 3 could perhaps be postponed to
3.4, when I expect including GSL functions should be on the TODO list.

jwe

[Jaroslav Hajek-2]

On Wed, Jul 16, 2008 at 11:24 PM, John W. Eaton <jwe@...> wrote:

> On 16-Jul-2008, Jaroslav Hajek wrote:
>
> | >  10. Improve compatibility of fsolve
> | >        - input/output args should be compatible
> | >        - use optimset for options
> | >
> |
> | I started to work on this, as well as adding/porting other functions
> | (fzero, lsqnonlin etc).
> | But I caught some rumors about 4. (moving optimization to OctaveFOrge
> | package) and I stopped until this is resolved. Personally, I'd vote
> | for having the basic nonlinear optimization/least squares/root finding
> | directly in Octave.
>
> I agree that Octave should have some general-purpose optimization
> and root finding functions included by default.
>
> | To be more constructive, I propose:
> |
> | 1. Update optimset to be reasonably Matlab-compatible (differences are
> | unavoidable due to different algorithms employed)
> |
> | 2. Port the 1-dimensional funcs fzero and fminbnd from octave-forge,
> | possibly update.
> |
> | 3a. Include the rest of MINPACK (currently used for fsolve) to
> | implement lsqnonlin. Also modify the MINPACK drivers to reverse
> | communication so as to make the functions reentrant.
> | OR
> | 3b. Implement fsolve and lsqnonlin drivers as m-files, possibly
> | wrapping the MINPACK step selection subprograms and reusing them.
> | More work, but probably a more maintainable and extendable outcome.
> |
> | 4. improve lsqnonneg using the QR updating funcs (they can also be
> | used for the Powell hybrid method in fsolve).
>
> These all seem OK to me.  If you are writing the code, then I think it
> is up to you to choose between 3a and 3b.  Does GSL have a C
> translation of the Minpack algorithms?  If so, maybe we could use
> those.  If we choose that, then item 3 could perhaps be postponed to
> 3.4, when I expect including GSL functions should be on the TODO list.
>

gsl/multifit and gsl/multiroot essentially are the MINPACK algorithms
rewritten in C
(multiroot also contains "plain" newton and damped newton), so here
the work for 3a has already been done.
I see, however, a couple drawbacks:
1. most of GSL uses hard-coded double as the real type. Now that we
have true single precision, this is really a big drawback. I don't
reckon there are plans to change this state in GSL. And I can't see
any good way out of this.
2. the "core" linear algebra operation of the MINPACK algorithms
(trust-region Levenberg-Marquardt and Powell's hybrid method) is the
QRP factorization. GSL has its own QRP code (as well as other linear
algebra codes) and employs it here. I think, however, that LAPACK is
fairly better.

If we intend to make Octave based on GSL (i.e. map Octave's functions
to GSL where possible), then these issues will necessarily arise.


> jwe
>


--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

[David Bateman-3]

Jaroslav Hajek wrote:

> I see, however, a couple drawbacks:
> 1. most of GSL uses hard-coded double as the real type. Now that we
> have true single precision, this is really a big drawback. I don't
> reckon there are plans to change this state in GSL. And I can't see
> any good way out of this.
> 2. the "core" linear algebra operation of the MINPACK algorithms
> (trust-region Levenberg-Marquardt and Powell's hybrid method) is the
> QRP factorization. GSL has its own QRP code (as well as other linear
> algebra codes) and employs it here. I think, however, that LAPACK is
> fairly better.

To me these are both good reasons not to use GSL. What I thought we were
gaining with GSL was

* upto date and maintained code
* code that generally performed better
* code that returned valid results for a wider range of input values.

Though given the two points above I'm not sure GSL is worth it.

D.

[Dmitri A. Sergatskov]

On Wed, Jul 16, 2008 at 5:04 PM, David Bateman <adb014@...> wrote:

> Jaroslav Hajek wrote:
>
>> I see, however, a couple drawbacks:
>> 1. most of GSL uses hard-coded double as the real type. Now that we
>> have true single precision, this is really a big drawback. I don't
>> reckon there are plans to change this state in GSL. And I can't see
>> any good way out of this.
>> 2. the "core" linear algebra operation of the MINPACK algorithms
>> (trust-region Levenberg-Marquardt and Powell's hybrid method) is the
>> QRP factorization. GSL has its own QRP code (as well as other linear
>> algebra codes) and employs it here. I think, however, that LAPACK is
>> fairly better.
>
> To me these are both good reasons not to use GSL. What I thought we were
> gaining with GSL was
>
> * upto date and maintained code
> * code that generally performed better
> * code that returned valid results for a wider range of input values.
>
> Though given the two points above I'm not sure GSL is worth it.
>

I guess one option is to lift the relevant code from GSL and adapt
it for octave data type and lapack. I think it is still better than carry
along the old Fortran code.

> D.
>

Dmitri.
--

[David Bateman]

Dmitri A. Sergatskov wrote:

> On Wed, Jul 16, 2008 at 5:04 PM, David Bateman <adb014@...> wrote:
>  
>> Jaroslav Hajek wrote:
>>
>>    
>>> I see, however, a couple drawbacks:
>>> 1. most of GSL uses hard-coded double as the real type. Now that we
>>> have true single precision, this is really a big drawback. I don't
>>> reckon there are plans to change this state in GSL. And I can't see
>>> any good way out of this.
>>> 2. the "core" linear algebra operation of the MINPACK algorithms
>>> (trust-region Levenberg-Marquardt and Powell's hybrid method) is the
>>> QRP factorization. GSL has its own QRP code (as well as other linear
>>> algebra codes) and employs it here. I think, however, that LAPACK is
>>> fairly better.
>>>      
>> To me these are both good reasons not to use GSL. What I thought we were
>> gaining with GSL was
>>
>> * upto date and maintained code
>> * code that generally performed better
>> * code that returned valid results for a wider range of input values.
>>
>> Though given the two points above I'm not sure GSL is worth it.
>>
>>    
>
> I guess one option is to lift the relevant code from GSL and adapt
> it for octave data type and lapack. I think it is still better than carry
> along the old Fortran code.

Ultimately it would be better if we could get the GSL maintainers to
reintegrate any such changes and support it themselves. Both float and
accelerated lapack libraries make sense for GSL to support and so I
think its probably a bad idea to bring this code into Octave itself.

D.
> de.
>  

[John W. Eaton]

On 16-Jul-2008, John W. Eaton wrote:

| On 16-Jul-2008, Søren Hauberg wrote:
|
| | I'm a bit curious, but why should the first 3.1.x release be called
| | 3.1.50? Are you so fond of the 2.1.50 release (the one that'll never go
| | away, it seems...) that you want a similar release in the 3.1.x
| | series :-)
|
| I want to make a series of snapshots before the next stable version is
| released.  Originally, I was planning to call the next major stable
| release 3.1.0, but if we do that, I'm not sure what version to use for
| the snapshots.  So I thought we could use 3.1.X for the snapshots and
| 3.2.0 for the next stable release.  I chose X == 50 because I think we
| are at least 50 percent of the way to 3.2.0.  :-)  But we can start
| with X at 0 if people think that makes more sense.

Oh, I could also make snapshots with version numbers like

  ss-20080717

but that might cause trouble for version comparison functions.  Does
it matter?

jwe

[David Bateman]

John W. Eaton wrote:

> On 16-Jul-2008, John W. Eaton wrote:
>
> | On 16-Jul-2008, Søren Hauberg wrote:
> |
> | | I'm a bit curious, but why should the first 3.1.x release be called
> | | 3.1.50? Are you so fond of the 2.1.50 release (the one that'll never go
> | | away, it seems...) that you want a similar release in the 3.1.x
> | | series :-)
> |
> | I want to make a series of snapshots before the next stable version is
> | released.  Originally, I was planning to call the next major stable
> | release 3.1.0, but if we do that, I'm not sure what version to use for
> | the snapshots.  So I thought we could use 3.1.X for the snapshots and
> | 3.2.0 for the next stable release.  I chose X == 50 because I think we
> | are at least 50 percent of the way to 3.2.0.  :-)  But we can start
> | with X at 0 if people think that makes more sense.
>
> Oh, I could also make snapshots with version numbers like
>
>   ss-20080717
>
> but that might cause trouble for version comparison functions.  Does
> it matter?
>
> jwe
>
>  

Yes that does indeed matter as then we'd have trouble preparing an
octave-forge release for 3.2 based on the 3.1.x development tree. I'd
prefer a version number 3.1.x where x can be whatever value you choose.

D.