Glib::IO->add_watch and reading the filehandle

[Matthew Braid-2]

Hi all,

I've just run into a little problem in one of my programs involving a
Glib::IO watch.

In order to allow the main program to run smoothly while Big Stuff(tm)
is happening, I've shifted a lot of time intensive work off to a
helper process which has been opened with IPC::Open2. Messages are
sent to the helper process and responses are picked up by the Glib::IO
watch. Usually this works fine with the following callback:

sub {
  my ($fno, $cond, $self) = @_;
  if ($cond & 'in' and
      sysread($self->{FromChild}, my $buff, 1024)) {
    $self->recvchunk($buff);
  }
  if ($cond & 'hup') {
    $self->recvchunk("\n");
    $self->{PipeOK} = 0;
    return 0;
  }
  return 1;
}

(to clarify, $self is passed in as data to the callback and is an
object that handles incoming and outgoing messages, translating as
needed. recvchunk is the sub that handles incoming blocks of data, and
$self->{FromChild} is the filehandle for the incoming messages).

Every so often it would stop working (ie, the helper process would
claim it sent a message, but the main process wouldn't act on it), and
after a lot of debugging it turned out that if an incoming message was
longer than 1024 characters the remaining part after the first 1024
characters wouldn't be handled until the _next_ message was sent - in
other words, if I sysread 1024 characters off a 1025
character-available stream, the Glib::IO watch doesn't still see the
file handle as readable, and so doesn't trigger the callback until
more data is pushed onto it. Unfortunately I can't use a while
(sysread(...)) in the callback cos then it hangs waiting for data.

As a workaround I upped the sysread to 10240 characters (which is
very, very large for a message in this system), but I'm pretty sure
this isn't Behaving As Expected, so I wanted to make sure I haven't
done something stupid in my callback.

Since I control both sides of the communication, I _can_ ensure that
the helper script never sends a message exactly 1024 characters in
length (empty lines are ignored, so sticking a newline at the end of a
1024 char message will work fine) and change my callback to something
like:

sub {
  my ($fno, $cond, $self) =@_;
  if ($cond & 'in') {
    while (1) {
      my $cnt = sysread($self->{FromChild}, my $buff, 1024);
      $self->recvchunk($buff);
      last if $cnt != 1024;
    }
  }
  if ($cond & 'hup') {
    $self->recvchunk("\n");
    $self->{PipeOK} = 0;
    return 0;
  }
  return 1;
}

but that only works _because_ I have full control on the incoming data
and isn't really a generic solution.

In case its important, messages sent back and forth are always
hashrefs that have been serialised into YAML and then Base64 encoded
and wrapped in '--- START MESSAGE---' and '---END MESSAGE---' lines,
so a full message is never a single unbroken line. Filehandles have
also been unbuffered with select.

TIA,
MB
_______________________________________________
gtk-perl-list mailing list
gtk-perl-list@...
http://mail.gnome.org/mailman/listinfo/gtk-perl-list

[muppet]

Matthew Braid wrote:

> Every so often it would stop working (ie, the helper process would
> claim it sent a message, but the main process wouldn't act on it), and
> after a lot of debugging it turned out that if an incoming message was
> longer than 1024 characters the remaining part after the first 1024
> characters wouldn't be handled until the _next_ message was sent - in
> other words, if I sysread 1024 characters off a 1025
> character-available stream, the Glib::IO watch doesn't still see the
> file handle as readable, and so doesn't trigger the callback until
> more data is pushed onto it. Unfortunately I can't use a while
> (sysread(...)) in the callback cos then it hangs waiting for data.

The trick in C code is to mark the file as non-blocking, and read chunks until
read returns 0 bytes.  I can't take the time to experiment right now, but i'm
reasonably certain that this works in perl, as well.

Unless you're using datagram sockets, there is no guarantee that an entire
message showed up at once, so you always have to be able to handle partial
reads.


--
muppet <scott at asofyet dot org>

_______________________________________________
gtk-perl-list mailing list
gtk-perl-list@...
http://mail.gnome.org/mailman/listinfo/gtk-perl-list

[Matthew Braid-2]

Hi,

2008/7/17 muppet <scott@...>:
> The trick in C code is to mark the file as non-blocking, and read chunks until
> read returns 0 bytes.  I can't take the time to experiment right now, but i'm
> reasonably certain that this works in perl, as well.

Gah. I'm still thinking in Perl/Tk terms. I was trying to figure out
why the filehandle wasn't being picked up as 'still readable' instead
of looking for a lower level solution. I've been doing a lot of head
slapping lately....

> Unless you're using datagram sockets, there is no guarantee that an entire
> message showed up at once, so you always have to be able to handle partial
> reads.

Partial reads aren't a problem (the main program just buffers the data
until it gets a whole message as long as the message so far is
structurally valid) it was just that sometimes that 'too big' message
wasn't followed by another message for a very long time, so the buffer
was never getting flushed.

If I use:

use Fcntl;
fcntl($FROM_CHILD, F_SETFL, fcntl($FROM_CHILD, F_GETFL, 0) | O_NONBLOCK);

that should do the trick.

Ta :)

MB
_______________________________________________
gtk-perl-list mailing list
gtk-perl-list@...
http://mail.gnome.org/mailman/listinfo/gtk-perl-list

[Kevin Ryde]

"Matthew Braid" <ptkperl@...> writes:
>
> opened with IPC::Open2

I've been using Proc::SyncExec because it's got good exec error
reporting.  Doesn't look like Open2/Open3 has that, you'd think they
probably should.

> I was trying to figure out why the filehandle wasn't being picked up
> as 'still readable'

Sounds like it should be right, if nobody else is doing a buffered read.
You could try you own select() in the io handler to see how it looks at
the point you would go back to the main loop.  Myself I've been using a
non-blocking pipe or socketpair and reading in the handler until
EWOULDBLOCK.

> use Fcntl;
> fcntl($FROM_CHILD, F_SETFL, fcntl($FROM_CHILD, F_GETFL, 0) | O_NONBLOCK);

"MYHANDLE->blocking(0)" might be easier (after loading IO::Handle or
whatever package it is that slips in that method).
_______________________________________________
gtk-perl-list mailing list
gtk-perl-list@...
http://mail.gnome.org/mailman/listinfo/gtk-perl-list