Feature Request 1572837: mutex API nesting level info

[Nathan Moore-5]

> Sure, when you write alone 100% of the application code and stay as the
> only maintainer and you write extensive documentation in the code you
> may use recursive mutexes and have not much problem, however if your
> work is shared or if someone (even yourself!) comes some time later to
> perform changes on the code, then recursive mutexes may appear as black
> magic.
>

>
> An excellent explanation about why to avoid recursive mutexes by someone
> that knows what he writes about:
>
> http://www.zaval.org/resources/library/butenhof1.html

Exactly the opposite because NutOs doesn't have preemptable threading.
That was all about using mutexes to protect memory-type resources, not IO
that can block.
In NutOs you protect memory resources with critical sections (turning off
interrupts) which
are the widest form of mutex (and sadly are recursive, which I will argue
against until I'm blue
in the face).   The only thing that can step on your memory resource usage
is an ISR unless
you throw NutThreadYields (directly or indirectly) into the middle of them.
His (from the link) arguments about holding mutexes for the shortest
possible time isn't as valid
on a system that doesn't have thread preemption since you effectively have
an "I'm the running
thread" mutex until you call the yield function anyway.

Recursive mutexes are very very good for IO routines, which do have
NutThreadYields in the
middle of them.
Say I have the code:

void print_record(FILE * f, struct record * r) {
  NutMutexLock(&(f->o_lock));
  fprintf(f, "%s : %i\t%f\n", r->name, r->count, r->percent);
  NutMutexUnlock(&(f->o_lock));
}

void print_status(FILE *f) {
  struct record * r;
  NutMutexLock(&(f->o_lock));
  fprintf(f, "Current status:\n");
  for (r = record_list; r; r = r->next) {
    print_record(f, r);
  }
  NutMutexUnlock(&(f->o_lock));
}

void print_record_named(char *n) {
   struct record *r;
   for (r = record_list, r, r = r->next) {
           if (0 == strcmp(n, r->name) ) {
              print_record(f, r);
           }
    }
}

Without the recursive mutex (which isn't actually in nut's FILE struct)
running code that uses these
(or any fprintf(f, ...) for that matter) could not guarentee that all of
their output was contiguous.
A higher priority thread could pop in there and write it's junk in the
middle of this output.
With the recursive mutex you can reuse component print functions and also
lock at the higher level
when you need to print other stuff together with those.

Nathan
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[duane ellis-6]

Bernard Fouché wrote:
> Hi,
>
> FYI, here is the point of view of eCos designers about recursive mutexes
> (taken from http://ecos.sourceware.org/docs-latest/ref/kernel-mutexes.html):
> ---
>  
POSIX threads solved this along time ago.  POSIX threads lets you many
types of MUTEXes.


    int pthread_mutex_init(pthread_mutex_t *mutex,
                const pthread_mutexattr_t *attr);

Where ATTRIBUTE is one of:
    NULL - you get some default

Or NOT NULL, and was specified via pthread_mutexattr_settype() as one of:

    PTHREAD_MUTEX_NORMAL
    PTHREAD_MUTEX_ERRORCHECK
    PTHREAD_MUTEX_RECURSIVE
    PTHREAD_MUTEX_DEFAULT

See:

http://www.opengroup.org/onlinepubs/007908799/xsh/pthread_mutexattr_settype.html

** Yes, NUT is not Pthreads.

But NUT can certainly learn from - pthreads. It is a well thought out
solution, that has passed the many eyeballs test.

One can learn from this.

-Duane.




_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Bernard Fouché]

Nathan Moore wrote:
>> An excellent explanation about why to avoid recursive mutexes by someone
>> that knows what he writes about:
>>
>> http://www.zaval.org/resources/library/butenhof1.html
>>    
>
> Exactly the opposite because NutOs doesn't have preemptable threading.
>  
I don't see why you restrict Buthenhof's remarks to preemptable
threading. As soon as you need a mutex, then, IMHO, his remarks have to
be considered. Otherwise, well, why would you need mutexes at all if
nothing can change the resource you are manipulating ? :)
> That was all about using mutexes to protect memory-type resources, not IO
> that can block.
>  
This is completely similar: I/O resources are seen as memory for the MCU
(I/O registers for instance, or a bit in a I/O register or in a
particular MCU register). Once again, Buthenof remarks are about design,
not low level details.
> In NutOs you protect memory resources with critical sections (turning off
> interrupts) which
> are the widest form of mutex (and sadly are recursive, which I will argue
> against until I'm blue
>  
Usually a CPU/MCU offers to turn on/off interrupts, without any nesting
level. Nested levels may exist in interrupt processing, but a particular
interrupt can only be turned off or on. You can't for instance, have the
following code:

1) turn interrupt X off
2) turn interrupt X off
3) turn interrupt X on
4) turn interrupt X on

As long as you reach step 3, interrup X is definitely turned on, so I
don't see how you can consider turning off/on interrupts being similar
to recursives mutexes.
> in the face).   The only thing that can step on your memory resource usage
> is an ISR unless
> you throw NutThreadYields (directly or indirectly) into the middle of them.
>  
In a preemptable OS, an ISR is used by the scheduler to eventually yield
the current thread and select a new one. An ISR in NutOs may wake up a
thread that was waiting for an I/O to complete (or any other event). So
basically the fact the OS is preemtable or not has no relation with
design guidelines linked to recursives mutexes, as you show yourself:

> Recursive mutexes are very very good for IO routines, which do have
> NutThreadYields in the
> middle of them.
> Say I have the code:
>
> void print_record(FILE * f, struct record * r) {
>   NutMutexLock(&(f->o_lock));
>   fprintf(f, "%s : %i\t%f\n", r->name, r->count, r->percent);
>   NutMutexUnlock(&(f->o_lock));
> }
>
> void print_status(FILE *f) {
>   struct record * r;
>   NutMutexLock(&(f->o_lock));
>   fprintf(f, "Current status:\n");
>   for (r = record_list; r; r = r->next) {
>     print_record(f, r);
>   }
>   NutMutexUnlock(&(f->o_lock));
> }
>
> void print_record_named(char *n) {
>    struct record *r;
>    for (r = record_list, r, r = r->next) {
>            if (0 == strcmp(n, r->name) ) {
>               print_record(f, r);
>            }
>     }
> }
>
> Without the recursive mutex (which isn't actually in nut's FILE struct)
> running code that uses these
> (or any fprintf(f, ...) for that matter) could not guarentee that all of
> their output was contiguous.
>  

That's exactly memory sharing as described by Buthenof. You would
'print' into a memory buffer or into a UART driven by I/O registed, the
problem is exactly the same.
> A higher priority thread could pop in there and write it's junk in the
> middle of this output.
> With the recursive mutex you can reuse component print functions and also
> lock at the higher level
> when you need to print other stuff together with those.
>  

This is an excellent example of a design problem: you are using the same
mutex for different needs.

1) You have a first level of locking which is 'lower' than the second.
This first level is in print_record() and needs a mutex to protect the
file descriptor being used to access the buffer that this file
descriptor uses.

2) You have a second level of usage in print_status(), which is 'higher'
than the first one, since you want to make sure that a bunch of callsto
print_record() are atomic. But you are also making direct usage of a
file descriptor in print_status (to print 'Current status:') which is
still another need.

All this mixed usages of a single mutex makes you think that recursive
mutexes are the solution.

Here is another way:

- have a 'lock_puts()' function:

void lock_puts(FILE * f, char *s) {
  NutMutexLock(&(f->o_lock));
  fputs(s,f);
  NutMutexUnlock(&(f->o_lock));
}

- everything that needs to print prepares a buffer in memory and sends
it to lock_puts().

No more recursive mutex needs, shortest locking time, the mutex is
clearly used only to protect the file descriptor and nothing else,
someone coming later and reading lock_puts() description understands
easily how to use it, and if ever you change of OS later, the same logic
will apply to your code base.

I would not have thought of a better example :)

Now you can object that what you want to print is so big that it can not
be first stored into memory before being sent out. But in that case it
would mean again that you have to change your application's design to
take that constraint into account. The solution would then to have only
'high level' function lock the mutex, and no more low level locking into
the function that directly print into the file descriptor: still no
recursive mutex needs.


_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Nathan Moore-5]

> This is completely similar: I/O resources are seen as memory for the MCU
> (I/O registers for instance, or a bit in a I/O register or in a
> particular MCU register). Once again, Buthenof remarks are about design,
> not low level details.



When I use memcpy(src, dest, len) on normal RAM from a thread on NutOs I
don't block the only thing that
can step on my operation is an ISR, but since I know that no ISRs are
supposed to be writing to src[] or dest[]
I don't need to protect them from ISRs with
NutEnterCritical/NutExitCritical.  Since I know that while another
thread might want to write to src or read/write dest under Linux I would get
a read lock on src and a write lock on
dest, but on NutOs these aren't needed because there is no preemptive
scheduling of threads or multiprocessing.
Relative to anything that should be able to change src[] or dest[] this
operation is atomic, so I don't use any form
of mutual exclusion other than knowing that this is the only running thread
right now.
Even the implementations of semaphores and the recursive mutexes take
advantage of knowing that the memory
that makes up the mutex current owner and lock count and the semaphore count
are not going to be trampled on
by someone else while operating on them.   They both contain a NutEvent too,
and that is protected with critical
sections because it uses the standard NutEvent operations which allow both
ISRs and application code to post
events.

>
> > In NutOs you protect memory resources with critical sections (turning off
> > interrupts) which
> > are the widest form of mutex (and sadly are recursive, which I will argue
> > against until I'm blue
> >
> Usually a CPU/MCU offers to turn on/off interrupts, without any nesting
> level. Nested levels may exist in interrupt processing, but a particular
> interrupt can only be turned off or on. You can't for instance, have the
> following code:

NutEnterCritical and NutExitCritical are designed to be able to be called
recurrsively by
pushing the current global interrupt enable flag onto the stack and then
disabling interrupts
to enter the critical and just pop it back off the stack for the exit.
I think the future implementation is to have a global counter that will keep
up with nesting levels
and enable interrupts when that counter is 0.  This is because compilers
can't understand when
applications mess with the stack directly.
I've argued against critical sections being recurrsive in the past.



> In a preemptable OS, an ISR is used by the scheduler to eventually yield
> the current thread and select a new one. An ISR in NutOs may wake up a
> thread that was waiting for an I/O to complete (or any other event). So
> basically the fact the OS is preemtable or not has no relation with
> design guidelines linked to recursives mutexes,


Awaking a thread with an event post from an ISR in NutOs only marks that
thread
to be moved to the run queue at the next call to NutThreadYield.  It does
not immediatly
wake up that thread, so it's not the same as preemptable multitasking.  If
the current running
thread that the ISR momentarily interrupted is in the middle of:
   some_flag = 1;
   while (some_flag) { /* nothing */ }
and the only place where some_flag is cleared is in another thread (perhapse
the newly awaken one)
then the current thread will run until the processor is reset.

> as you show yourself:
>
> > Recursive mutexes are very very good for IO routines, which do have
> > NutThreadYields in the
> > middle of them.
> > Say I have the code:
> >
> > void print_record(FILE * f, struct record * r) {
> >   NutMutexLock(&(f->o_lock));
> >   fprintf(f, "%s : %i\t%f\n", r->name, r->count, r->percent);
> >   NutMutexUnlock(&(f->o_lock));
> > }
> >
> > void print_status(FILE *f) {
> >   struct record * r;
> >   NutMutexLock(&(f->o_lock));
> >   fprintf(f, "Current status:\n");
> >   for (r = record_list; r; r = r->next) {
> >     print_record(f, r);
> >   }
> >   NutMutexUnlock(&(f->o_lock));
> > }
> >
> > void print_record_named(char *n) {
> >    struct record *r;
> >    for (r = record_list, r, r = r->next) {
> >            if (0 == strcmp(n, r->name) ) {
> >               print_record(f, r);
> >            }
> >     }
> > }
> >
> > Without the recursive mutex (which isn't actually in nut's FILE struct)
> > running code that uses these
> > (or any fprintf(f, ...) for that matter) could not guarentee that all of
> > their output was contiguous.
> >
> That's exactly memory sharing as described by Buthenof. You would
> 'print' into a memory buffer or into a UART driven by I/O registed, the
> problem is exactly the same.

No matter how big you make your buffers I can come up with a reasonable
example that would need a bigger buffer to work correctly.
Also I might only print something that big very rarely, and my example
showed
that the size of what I'm printing is not deterministic.  The list may be
empty or
the list itself may take up half of the 128K of ram I have attached to my
AVR
(less is actually directly addressable and for the stdio f()s it can't be in
paged
ram.

>
> > A higher priority thread could pop in there and write it's junk in the
> > middle of this output.
> > With the recursive mutex you can reuse component print functions and also
> > lock at the higher level
> > when you need to print other stuff together with those.
> >
>
> This is an excellent example of a design problem: you are using the same
> mutex for different needs.
>
> 1) You have a first level of locking which is 'lower' than the second.
> This first level is in print_record() and needs a mutex to protect the
> file descriptor being used to access the buffer that this file
> descriptor uses.
>
> 2) You have a second level of usage in print_status(), which is 'higher'
> than the first one, since you want to make sure that a bunch of callsto
> print_record() are atomic. But you are also making direct usage of a
> file descriptor in print_status (to print 'Current status:') which is
> still another need.
>
> All this mixed usages of a single mutex makes you think that recursive
> mutexes are the solution.

Well that's exactly what they are good for.  You have not told me why they
are bad, only
that I'm doing stuff wrong.  I don't see anything wrong with using a
recursive mutex.

>
>
> Here is another way:
>
> - have a 'lock_puts()' function:
>
> void lock_puts(FILE * f, char *s) {
>  NutMutexLock(&(f->o_lock));
>  fputs(s,f);
>  NutMutexUnlock(&(f->o_lock));
> }
>
> - everything that needs to print prepares a buffer in memory and sends
> it to lock_puts().
>
> No more recursive mutex needs, shortest locking time, the mutex is
> clearly used only to protect the file descriptor and nothing else,
> someone coming later and reading lock_puts() description understands
> easily how to use it, and if ever you change of OS later, the same logic
> will apply to your code base.

Ok, implement a splice with your method.
I want to take stuff from Uart1 and output it to Uart2 up until an IO line
goes low.
When that happens I want Uart1 to be used by other threads to output any
status
messages that they may have until that IO line goes high again (which will
likely
cause an interrupt which will post an event).


Nathan
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Bernard Fouché]

Nathan Moore wrote:
> Well that's exactly what they are good for.  You have not told me why they
> are bad, only
> that I'm doing stuff wrong.  I don't see anything wrong with using a
> recursive mutex.
>  

That's exactly the point: they are bad because they encourage you to do
wrong stuff! Without recursive mutexes you would have reconsidered your
design and discovered that you can live without them. However for quick
prototyping they may be handy but then you have to work twice on the
same code, once for writing the prototype, once to finalize the design.

> Ok, implement a splice with your method.
> I want to take stuff from Uart1 and output it to Uart2 up until an IO line
> goes low.
> When that happens I want Uart1 to be used by other threads to output any
> status
> messages that they may have until that IO line goes high again (which will
> likely
> cause an interrupt which will post an event).
>
>  

I think that unless real code is written like you did in your first
example, it's difficult to provide a good answer:  in this second
problem description, there is no specification forbidding to always
allow threads to output to UART1, even if UART1's input is to be output
on UART2 according to the IO line (there is also nothing in your
description about what to do with UART1's input when the IO line changes
of state and threads are outputing to UART1). Maybe no mutex is needed
at all.

Note that such a splice can be difficult: if the IO line drives the
application behavior regarding UART's, you may reach a situation where
the IO line changes of state but there is already a character placed in
a UART register (rx and/or tx). So you may send/receive unexpected
characters on a UART, just after the IO line state change. (This problem
is not mutex related at all)

That's is why there is often 'lag' when using hardware handshaking in a
serial link: if CTS or RTS changes of state, one or more characters may
still reach the remote party because some UART registers were set up
with data before the detection of the change of state.

Maybe multiplexing would be a better solution? This would allow you to
specify what is the final usage of a character and helps the remote
party not to badly mix received characters.
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Nathan Moore-5]

On Tue, Jul 22, 2008 at 12:58 PM, Bernard Fouché <bernard.fouche@...>
wrote:

> Nathan Moore wrote:
> > Well that's exactly what they are good for.  You have not told me why
> they
> > are bad, only
> > that I'm doing stuff wrong.  I don't see anything wrong with using a
> > recursive mutex.
> >
>
> That's exactly the point: they are bad because they encourage you to do
> wrong stuff! Without recursive mutexes you would have reconsidered your
> design and discovered that you can live without them. However for quick
> prototyping they may be handy but then you have to work twice on the
> same code, once for writing the prototype, once to finalize the design.

I can live without lots of things.  I can live without a compiler if I have
to.
I can live without a text editor and just do
   $ cat > source.c
   #include ...

but that's not something I do very regularly.



>
> > Ok, implement a splice with your method.
> > I want to take stuff from Uart1 and output it to Uart2 up until an IO
> line
> > goes low.
> > When that happens I want Uart1 to be used by other threads to output any
> > status
>
CORRECTION -- ... I want Uart2 to be used by other threads to ...

>
> > messages that they may have until that IO line goes high again (which
> will
> > likely
> > cause an interrupt which will post an event).
> >
> >
>
> I think that unless real code is written like you did in your first
> example, it's difficult to provide a good answer:  in this second
> problem description, there is no specification forbidding to always
> allow threads to output to UART1, even if UART1's input is to be output
> on UART2 according to the IO line (there is also nothing in your
> description about what to do with UART1's input when the IO line changes
> of state and threads are outputing to UART1). Maybe no mutex is needed
> at all.

Sorry about the mistake in my example.

>
>
> Note that such a splice can be difficult: if the IO line drives the
> application behavior regarding UART's, you may reach a situation where
> the IO line changes of state but there is already a character placed in
> a UART register (rx and/or tx). So you may send/receive unexpected
> characters on a UART, just after the IO line state change. (This problem
> is not mutex related at all)

It doesn't have to be perfect, just keep messages together.  If a status
message
is being output it should be output completely and contiguously.


>
>
> Maybe multiplexing would be a better solution? This would allow you to
> specify what is the final usage of a character and helps the remote
> party not to badly mix received characters.

I do not understand this.  The second sentence doesn't make sense.

Nathan
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[duane ellis-6]

I've not been following this entire conversation about recursive mutexes
in detail.

 >> [about recursive mutexes]

 >> That's exactly the point: they are bad because they encourage you to
do  the wrong stuff

There is nothing wrong with recursive mutexes.

There is an old saying:
        If your only tool is a hammer, everything is a nail.

I have smashed my fingers with that proverbial hammer a few times.  
Sometimes - it was the only type of nail I had to build my project.
Sometimes - I did  not know any better - and I learned. Smashing fingers
hurt. At times I decided to *never* use that type of nail again. Over
time - I learned how to use that type of nail properly and changed my
mind. In your home, I bet there are over 20 different types and sizes of
nails. The skilled builder who built your home knew when and where to
use each type of nail. (let us hope he was a skilled builder!)

Mutexes are like nails they are building tools, a skilled
craftsmen(person) knows how to choose right one.

The tool box (ethernut) is missing a few other types of nails that
others need.

-------

About the "David Butenhof" - article that somebody linked to:

    http://www.zaval.org/resources/library/butenhof1.html

his basic thesis is this:

     Recursive locks are so easy - to get wrong - that you should never
use them.
     People are lazy - and do not properly study the situation at hand
and choose badly.

Put another way:
 
    It is easy to smash your finger with that type of nail.
    So - NEVER use that type of nail.
    Period. Full Stop.  End of Story.
    There should be laws against that type of nail.

You know what?

    A scalpel (surgeons knife) has the same property. Either it is in
the hands of an idiot, or a skilled surgeon.

As David Butenhof says:
 
    Recursive mutexes are a hack. There's nothing wrong
    with using them, but they're a crutch. Got a broken leg
    or library? Fine, use the crutch. But at least be aware
    that you're using a crutch, and why;

yes, he goes on, I cut him off - he mentions a a doctor.. To paraphrase
him a bit -

    Maybe - given the situation at hand...
        recursive mutexes are exactly what the skilled surgeon ordered.

I like to think most of us are of the "skilled surgeon" type.

-Duane.


_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Bernard Fouché]

Nathan Moore wrote:

>
>
> I can live without lots of things.  I can live without a compiler if I have
> to.
> I can live without a text editor and just do
>    $ cat > source.c
>    #include ...
>
> but that's not something I do very regularly.
>
>
>  

This discussion leading nowhere, I think I can live without it also.

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion