[RFC] octave-to-backend notification of property change

[Michael Goffioul-2]

Hi all,

I'm currently in the process of implementing the mechanism used by
octave to notify the backend about property changes that can impact
GUI elements. I'm not talking here about objects contained in an axes
system, which are handled during the axes rendering, but about
actual GUI elements. For instance:
- "name" in figure => change figure window title bar
- "string" in uicontrol => change control string (for instance the
text of a button)

When such properties are changed, the backend must be notified
in order to update its internal GUI elements. I thought about 2
possible mechanisms:

1) In the "set" method of the corresponding property (for instance
in figure::properties::set_name), call a specific method in the
backend to notify it (for instance update_figure_name). The
real backend implementation can then overload the method and
implement the appropriate action.

2) Delay the backend notification until next "drawnow". This can be
done by adding a "modified" flag to the property class, which is set
to "true" when a property is modified. During the next drawnow, the
real backend can then poll all properties it knows about, check if
they changed and implement the appropriate action.

Now let's talk about advantages and drawbacks. Option 1 imply
a lot of methods to add to the backend classes, especially for
uicontrol objects, but the effort can be largely reduced with some
preprocessor magic (I experimented it). OTOH the backend becomes
very responsive, because the it gets notified almost immediately.
However for functions/scripts that modify a lot of those properties,
the CPU overhead due to method call and possibly thread
synchronization can quicly slow things down. At the opposite,
option 2 does not requires much coding in octave (most of the
work will be done by the backend), but you'll only see the visual
result of changing a property in the next drawnow.

I experimented a little in the competition, and used the following
piece of code

figure;
for k=1:500, set(gcf, 'position', [k 50 500 500]), end

You can see the figure window moving, which means that something
similar to option 1 is used. I started to implement option 1, but then
I started to doubt whether it was the best solution. I tried to figure
out a use-case where a bahevior like in option 1 would be required,
but could not find any.

So, does anybody have any comment or scenario that would be
in favor of one option vs. the other?

Thanks.
Michael.

[David Bateman]

Does the drawnow in most cases only happen when you come back to the
prompt. In that case option 2 would be completely unresponsive unless
the user included the drawnows in their program. Also your example

figure;
for k=1:500, set(gcf, 'position', [k 50 500 500]), end

running in th ehg head with the fltk backend resulted in the figure in
the initial state in the upper right corner of the screen and then
instantly in a final state without passing through any intermediate
states, which seems to confirm that the drawnow is only at the return to
the prompt.

Regards
David

--
David Bateman                                David.Bateman@...
Motorola Labs - Paris                        [phone] (Ph)
Parc Les Algorithmes, Commune de St Aubin    [phone] (Mob)
91193 Gif-Sur-Yvette FRANCE                  [phone] (Fax)

The information contained in this communication has been classified as:

[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary

[Michael Goffioul-2]

On Fri, Jul 4, 2008 at 3:06 PM, David Bateman
<David.Bateman@...> wrote:
> Does the drawnow in most cases only happen when you come back to the
> prompt. In that case option 2 would be completely unresponsive unless
> the user included the drawnows in their program.

The competition documentation states that the graphics event queue is
automatically flushed when octave prompt returns or when the following
functions are called: drawnow, input, pause, getframe, ginput, waitfor
(and maybe a few others I don't remember). So basically, the queue is
flushed just before the user can potentially interact again.

So indeed, option 2 is a matter of responsiveness. OTOH, do you see
a use-case when such unresponsiveness is not acceptable? Axes
rendering for instance does not occur until the next drawnow anyway.

> Also your example
>
> figure;
> for k=1:500, set(gcf, 'position', [k 50 500 500]), end
>
> running in th ehg head with the fltk backend resulted in the figure in
> the initial state in the upper right corner of the screen and then
> instantly in a final state without passing through any intermediate
> states, which seems to confirm that the drawnow is only at the return to
> the prompt.

The octave-to-backend notification I experimented (option 1) is not
in hg head, so the observed behavior is normal. There is also a
specific behavior in FLTK backend in the sense that the GUI event
loop is run synchronously with octave thread, within input_event_hook
handler. So the GUI is completely frozen (no event dispatching) when
octave is not waiting for user input.

Michael.

[Maciej Gajewski]

I have and idea which is hybrid of both of these solutions:

Let all "set" methods notify backend using the same method, with signature
like this:

virtual void base_graphics_backend::property_changed( const graphics_handle&
owner, const std::string& name );

This would be desd simple at part of octave, and allows abckend to choose  how
to deal with it.

FLTK backend, which lives in octave's main thread, could filter our important
properties and repaint if needed.

My Qt backend[1], which uses separate thread for GUI, could implement this
method so it sends cross-thread messages to GUI when important property
changes.

Additionally, backend couild decide which properties should triger immediate
reaction, and which can wait to next repaint.

To make things fancier - backend (both single-and multi threaded) could put
(handle, property) tuple into some sort of queue, and repaint only at some
sane intervals. This could avoid overheting CPU with frequent redrawing.

Regarding competition: I'm pretty sure that current implementation is
multi-threaded - so described behavior with moving figure. 'drawnow' is
possibly just a relic.

[1] Qt backent is very actively developed by me, see:
http://code.google.com/p/qoplot/

> Hi all,
>
> I'm currently in the process of implementing the mechanism used by
> octave to notify the backend about property changes that can impact
> GUI elements. I'm not talking here about objects contained in an axes
> system, which are handled during the axes rendering, but about
> actual GUI elements. For instance:
> - "name" in figure => change figure window title bar
> - "string" in uicontrol => change control string (for instance the
> text of a button)
>
> When such properties are changed, the backend must be notified
> in order to update its internal GUI elements. I thought about 2
> possible mechanisms:
>
> 1) In the "set" method of the corresponding property (for instance
> in figure::properties::set_name), call a specific method in the
> backend to notify it (for instance update_figure_name). The
> real backend implementation can then overload the method and
> implement the appropriate action.
>
> 2) Delay the backend notification until next "drawnow". This can be
> done by adding a "modified" flag to the property class, which is set
> to "true" when a property is modified. During the next drawnow, the
> real backend can then poll all properties it knows about, check if
> they changed and implement the appropriate action.
>
> Now let's talk about advantages and drawbacks. Option 1 imply
> a lot of methods to add to the backend classes, especially for
> uicontrol objects, but the effort can be largely reduced with some
> preprocessor magic (I experimented it). OTOH the backend becomes
> very responsive, because the it gets notified almost immediately.
> However for functions/scripts that modify a lot of those properties,
> the CPU overhead due to method call and possibly thread
> synchronization can quicly slow things down. At the opposite,
> option 2 does not requires much coding in octave (most of the
> work will be done by the backend), but you'll only see the visual
> result of changing a property in the next drawnow.
>
> I experimented a little in the competition, and used the following
> piece of code
>
> figure;
> for k=1:500, set(gcf, 'position', [k 50 500 500]), end
>
> You can see the figure window moving, which means that something
> similar to option 1 is used. I started to implement option 1, but then
> I started to doubt whether it was the best solution. I tried to figure
> out a use-case where a bahevior like in option 1 would be required,
> but could not find any.
>
> So, does anybody have any comment or scenario that would be
> in favor of one option vs. the other?
>
> Thanks.
> Michael.

[Michael Goffioul-2]

On Fri, Jul 4, 2008 at 8:18 PM, Maciek Gajewski
<maciej.gajewski0@...> wrote:
> I have and idea which is hybrid of both of these solutions:
>
> Let all "set" methods notify backend using the same method, with signature
> like this:
>
> virtual void base_graphics_backend::property_changed( const graphics_handle&
> owner, const std::string& name );

I also considered that possibility, but I found it way less efficient than
having specific methods

virtual void base_graphics_backend::figure_property_changed(...)

Indeed, from the octave point of view, when such notification method
is called, it knows the object and property (because it's called from
the set_xxx handler), so it can call a specific method directly.
OTOH, using generic signature as you propose would just force
the backend to re-do a lot of string comparisons to know which
property was changed.

Note that using specific notifiers does not prevent the backend
from delaying its reaction.

Michael.

[Maciej Gajewski]

Michael Goffioul write:

> > Let all "set" methods notify backend using the same method, with
> > signature like this:
> >
> > virtual void base_graphics_backend::property_changed( const
> > graphics_handle& owner, const std::string& name );
>
> I also considered that possibility, but I found it way less efficient than
> having specific methods
>
> virtual void base_graphics_backend::figure_property_changed(...)
>
> Indeed, from the octave point of view, when such notification method
> is called, it knows the object and property (because it's called from
> the set_xxx handler), so it can call a specific method directly.
> OTOH, using generic signature as you propose would just force
> the backend to re-do a lot of string comparisons to know which
> property was changed.

So, do you want to have notifier method for each property? That would require
backend author to re-implement a whole lot of methods, most of which would be
almost identical.

String comparisons are cheap these days. I could event bet, that entire
processor time used for these comparisons, on all octave instances, all
around the world, would be shorter that time spent by programmers to code and
maintain all these methods :)

Besides - these notifiers would be used in interactive environments, when
performance isn't all that important as in - for example - running large
batch calculations.

At the other side - it could be good to have sepearate notifiers per object
class:

void figure_property_changed( graphics_handle fh,  std::string prop_name );
void axes_property_changed( graphics_handle ah,  std::string prop_name );
void axes_item_property_changed( graphics_handle ih,  std::string prop_name );
void uiobject_property_changed( graphics_handle uh,  std::string prop_name );

Maciek Gajewski

[Michael Goffioul-2]

On Fri, Jul 4, 2008 at 10:55 PM, Maciek Gajewski
<maciej.gajewski0@...> wrote:
> So, do you want to have notifier method for each property? That would require
> backend author to re-implement a whole lot of methods, most of which would be
> almost identical.

This would indeed mean a lot of small methods, but they wouldn't be almost
identical: changing figure's position or name, or uicontrol's string lead to
different actions in the GUI. Moreover, using simple preprocessor magic,
I could easily reduce the coding effort in the backend to something like

DEFUN_UPDATER(figure, position, ...)
{
  // update figure window location
}

which is as easy as adding another "else if" part to an existing if
statement.

> String comparisons are cheap these days. I could event bet, that entire
> processor time used for these comparisons, on all octave instances, all
> around the world, would be shorter that time spent by programmers to code and
> maintain all these methods :)

Even if string comparison is cheap, that's not a reason to waste CPU
cycles when you can avoid it. What I find stupid in this case is not
using an information that you have, forcing the backend to find it
back and wasting CPU time.

> Besides - these notifiers would be used in interactive environments, when
> performance isn't all that important as in - for example - running large
> batch calculations.

I don't agree with this. If you consider a resize callback of a figure
containing some complex GUI with a lot of controls, you can
end up with updating a lot of things in the GUI. And if these kind
of things takes 5x longer than in the competition, you can be sure
people will start complaining :-)

> At the other side - it could be good to have sepearate notifiers per object
> class:
>
> void figure_property_changed( graphics_handle fh,  std::string prop_name );
> void axes_property_changed( graphics_handle ah,  std::string prop_name );
> void axes_item_property_changed( graphics_handle ih,  std::string prop_name );
> void uiobject_property_changed( graphics_handle uh,  std::string prop_name );

That's indeed a minimum. And I almost started to implement that idea.
But I found out that the coding overhead with specific notifiers could be
largely reduced with some preprocessor statements and that a more
generic approach as above was as complex as using specific
notifiers, but with additional (avoidable) string comparisons.

Anyway, the original question was whether the backend should be
notified immediately on all property change, or everything should be
delayed until the next drawnow. From the first reactions, it seems that
early notification of the backend (in whatever form) is preferred.

Michael.

[Maciej Gajewski]

Michael Goffioul wrote:
> This would indeed mean a lot of small methods, but they wouldn't be almost
> identical: changing figure's position or name, or uicontrol's string lead
> to different actions in the GUI. Moreover, using simple preprocessor magic,
> I could easily reduce the coding effort in the backend to something like
>
> DEFUN_UPDATER(figure, position, ...)
> {
>   // update figure window location

It all depends what are expecting to get.

If you want react only to few events, like: figure resizing, figure title
change, ui control resizing and label change etc, that's fine.

But it would be really great to have consistent notification mechanism, which
could replace current 'redraw_figure' completely. I'd love to have interface
like this:

virtual void property_changed ( handle, name );
virtual void object_created ( handle );
virtual void object_deleted ( handle );

This would allow me to react selectively to all events, without need to
re-create entire GUI objects tree each time redraw_figure() is called.

In my backend I have separate GUI object for each octave's graphic object. So
property_changed would act like hub, identifying appropriate GUI object by
it's handle and sent cross-thread notification to it.

If we have each method called for each property, it would be like over 200
separate notifiers! And event more after implementing uicontrols.

And in multi-threaded backend they would all do the same: send message to GUI
thread to update appropriate control.

Just imagine the hell of implementing and then maintaining it: when new
property is added, all backends had to be updated.

> Even if string comparison is cheap, that's not a reason to waste CPU
> cycles when you can avoid it. What I find stupid in this case is not
> using an information that you have, forcing the backend to find it
> back and wasting CPU time.
>
> > Besides - these notifiers would be used in interactive environments, when
> > performance isn't all that important as in - for example - running large
> > batch calculations.
>
> I don't agree with this. If you consider a resize callback of a figure
> containing some complex GUI with a lot of controls, you can
> end up with updating a lot of things in the GUI. And if these kind
> of things takes 5x longer than in the competition, you can be sure
> people will start complaining :-)

It definitely wouldn't take 5x longer because of few short string comparison.
All this drawing stuff, talking to window system (whichever window system it
would be), obtaining resources required for drawing etc. is so much larger
job that comparing 200 short strings. Besides we are required to compare
strings anyway when reading radio properties.

And event when string comparison will become problem, efficient hash table
could fight it easily.

Another idea: create large enum in each property set, containing numerical
identifiers of all properties. I think this is doable using awk script. Then
limit using strings only in most top-level user-interacting functions, and
use numerical id everywhere else.

But honestly I think this is an overkill. I'd stay with string names.

Maciek Gajewski

[Michael Goffioul-2]

On Sat, Jul 5, 2008 at 3:50 PM, Maciek Gajewski
<maciej.gajewski0@...> wrote:

>> DEFUN_UPDATER(figure, position, ...)
>> {
>>   // update figure window location
>
> It all depends what are expecting to get.
>
> If you want react only to few events, like: figure resizing, figure title
> change, ui control resizing and label change etc, that's fine.
>
> But it would be really great to have consistent notification mechanism, which
> could replace current 'redraw_figure' completely. I'd love to have interface
> like this:
>
> virtual void property_changed ( handle, name );
> virtual void object_created ( handle );
> virtual void object_deleted ( handle );
>
> This would allow me to react selectively to all events, without need to
> re-create entire GUI objects tree each time redraw_figure() is called.
>
> In my backend I have separate GUI object for each octave's graphic object. So
> property_changed would act like hub, identifying appropriate GUI object by
> it's handle and sent cross-thread notification to it.
>
> If we have each method called for each property, it would be like over 200
> separate notifiers! And event more after implementing uicontrols.
>
> And in multi-threaded backend they would all do the same: send message to GUI
> thread to update appropriate control.

The idea here is not to have such notifier for all objects and all
properties, but only for those that are *not*¨part of the rendering
process. That means only objects/properties related to GUI
elements. In the current graphics code, this only concerns
very few properties in figure class.

All the rest is handled by the rendering process in drawnow.
It wouldn't make sense to have notifiers for these properties,
because the only reaction is to redraw the figure content.
This is already handled by the "drawnow" function, which
checks if a figure has been modified and asks the backend
to redraw it if necessary.

>From what you say above, it seems that you're trying to
maintain in the GUI thread an object tree that is the image
of the object tree in octave, and that you would like to have
notifiers for all objects/properties in order to maintain your
image tree in sync with octave. This does not fit with the
current design in the graphics code, where the intention is
to have a backend to use directly the octave objects
(hence the required locking mechanism).

Michael.

[Maciej Gajewski]

Michael Goffioul wrote:
> The idea here is not to have such notifier for all objects and all
> properties, but only for those that are *not*¨part of the rendering
> process. That means only objects/properties related to GUI
> elements. In the current graphics code, this only concerns
> very few properties in figure class.

>
> From what you say above, it seems that you're trying to
> maintain in the GUI thread an object tree that is the image
> of the object tree in octave, and that you would like to have
> notifiers for all objects/properties in order to maintain your
> image tree in sync with octave. This does not fit with the
> current design in the graphics code, where the intention is
> to have a backend to use directly the octave objects
> (hence the required locking mechanism).

Below I'll try to explain how performance can be greatly improved with
notifications, how dividing properties on "gui-related"
and "rendering-related" could harm backend development, and how - using
modern open source technologies - octave could outperform competition.

Let's consider following object tree (matches attached screenshot):

figure
  - axes1
    * image1
    * line1
    * text1
  - axes2
    * line2

On the figure we have two top level axes objects. They don't overlap, so there
is no need for backend to redraw entire figure when some small property
changes. If I type
 # set( line2, 'Marker', 'o' );
in octave prompt, effectively implemented backend should repaint only affected
axes (axes2 in this example).

But optimization should go event further: lets do:
 # set( text1, 'FontWeight', 'bold');

Re-rendering entire axes could be very expensive, i.e. because of large image
object. Converting large 3-D cdata into RGB pixmap is time-consuming. Good
implementation would convert cdata provided by octave into rendering
system-specific entity (be it Windows DDB, X11 pixmap or OpenGL texture), and
use it for painting as long as cdata and other appropriate properties remains
unchanged.

In this case, changing font weight of text item should force backend only to
repaint part of image directly below changed text, using cached converted
data. This not only reduces area to repaint by factor of 100, but also
re-uses already prepared data.

To achieve this, backend needs to know when property changes, and refresh only
affected part of the tree.

As I understand, you want to divide graphics objects into 'GUI objects'
(figure, uicontrols in future?), and 'rendered objects'. This isn't bad idea
to design backend around this separation. As I understand - this is the way
FLTK backend is designed - it uses FLTK for GUI objects, and glrenderer to
render axes and all its objects.

But I can't understand why backend developers (*ehem* me *ehem*) have to be
forced to use this scheme.

As you noted - I'm using tree of visual objects that mirrors octave's internal
object tree. This isn't in contradict with using octave objects directly.
I use (or: I will use once they become thread-safe) octave objects directly as
property containers, but to draw them, I have to use objects provided by
tookit and graphics library.

FLTK backend will have to use the same scheme once uicontrols arrive. It will
propably have to implement uicontrols using FLTK widgets . The only
difference is in fact that Qt backend goes deeper using Qt's 'Graphics View'
framework ( http://doc.trolltech.com/4.4/graphicsview.html ). This framework
allows for every line, text, image to be represented and painted as separate
object, with smart and partial repaints, optimizations, full user interaction
and all goodies.

So what I am asking for is a simple yet robust notification scheme which would
allow me to keep backend in sync with octave's object tree. This way
Qt-backend could instantly react to property changes, be responsive, and -
eventually - perform better and look better that competition's product.

Just three methods would do the magic:
- property_changed( handle, name )
- object_added( handle )
- object_removed( handle )

They don't collide with 'figure_title_changed' or 'figure_moved' proposed
earlier, but, IMHO, make them superfluous (again the string comparision and
stuff).

Maciek Gajewski

[Michael Goffioul-2]

On Sun, Jul 6, 2008 at 11:57 AM, Maciek Gajewski
<maciej.gajewski0@...> wrote:
> But I can't understand why backend developers (*ehem* me *ehem*) have to be
> forced to use this scheme.

That's a weird statement: "octave's design does not fit well with my code
and I don't see why I should be forced to use octave's design"...

> Just three methods would do the magic:
> - property_changed( handle, name )
> - object_added( handle )
> - object_removed( handle )

Come with an appropriate changeset and we'll discuss it further.

Michael.