Re: svn commit: r647384 - in /apr/apr/trunk: CHANGES include/apr_pools.h memory/unix/apr_pools.c

[Joe Orton]

On Sat, Apr 12, 2008 at 06:36:15AM -0000, Mladen Turk wrote:
> Author: mturk
> Date: Fri Apr 11 23:36:14 2008
> New Revision: 647384
>
> URL: http://svn.apache.org/viewvc?rev=647384&view=rev
> Log:
> Introduce apr_pool_create_core_ex

What problem is this solving?

"This function should be used for short living pools, usually ones that
are created and destroyed either in a loop or inside function call."

I don't understand that at all.  Needing to use an unparented pool
sounds like an application bug/design issue.  Can you explain the
context for this?

joe

[mturk]

Joe Orton wrote:
 > On Sat, Apr 12, 2008 at 08:42:59AM -0000, Mladen Turk wrote:
 >> Author: mturk
 >> Date: Sat Apr 12 01:42:51 2008
 >> New Revision: 647394
 >>
 >> URL: http://svn.apache.org/viewvc?rev=647394&view=rev
 >> Log:
 >> Introduce apr_pool_sys_allocator_set
 >
 > Why can't you just use a custom allocator for whatever the problem is
here?  That's what the allocator abstraction is *for*, right?  Adding
yet another abstraction above the allocator seems really wrong;
especially since it introduces a bunch of global state and *everybody*
suffers the overhead of the additional function calls and conditionals.
 >

How can I use custom allocator from outside apr?
The allocator is private, so cannot be customized from application.
It relies on malloc/free, and the point of patch is to abstract those
calls. Patch itself only sets the function pointer, and since its
warper is inline function the proper compiler should make no
overhead except one JNZ instruction.

The point is that apr internally uses system allocator and
we presume this should be malloc/free. Any other mechanism
is ruled out by that presumption. malloc/free might not be the
most effective way to allocate the memory from the system in
all use cases. It also doesn't allow things like profiling
and all other sort of things the application might have.
Once you create an allocator you don't have a track what's
going on beneath. You don't know how apr internally allocates
the memory from the system. Those callbacks if set are the
solution.

However if you really think this is an too high overhead
we might add compile time switch (APR_POOL_PROFILE) that
if not set will skip out this single compare instruction.

Regards
--
(TM)

[Joe Orton]

On Tue, Apr 15, 2008 at 06:48:49AM +0200, Mladen Turk wrote:
> Joe Orton wrote:
> > On Sat, Apr 12, 2008 at 08:42:59AM -0000, Mladen Turk wrote:
> >> Author: mturk
> >> Date: Sat Apr 12 01:42:51 2008
> >> New Revision: 647394
> >>
> >> URL: http://svn.apache.org/viewvc?rev=647394&view=rev
> >> Log:
> >> Introduce apr_pool_sys_allocator_set

Mladen - I think you posted that response to this thread as well as the
other; this thread was about r647384 (apr_pool_create_core_ex) not
r647394 (apr_pool_sys_allocator_set).

joe

[Sander Striker-2]

On Wed, Apr 16, 2008 at 3:52 PM, Joe Orton <jorton@...> wrote:

> On Tue, Apr 15, 2008 at 06:48:49AM +0200, Mladen Turk wrote:
>> Joe Orton wrote:
>> > On Sat, Apr 12, 2008 at 08:42:59AM -0000, Mladen Turk wrote:
>> >> Author: mturk
>> >> Date: Sat Apr 12 01:42:51 2008
>> >> New Revision: 647394
>> >>
>> >> URL: http://svn.apache.org/viewvc?rev=647394&view=rev
>> >> Log:
>> >> Introduce apr_pool_sys_allocator_set
>
> Mladen - I think you posted that response to this thread as well as the
> other; this thread was about r647384 (apr_pool_create_core_ex) not
> r647394 (apr_pool_sys_allocator_set).

I checked on the archives, as I stumbled over the code and wondered
what the heck it was for.  I don't seem to find a response as to what
this functionality is actually needed for.  In case I missed it, could you
point me to it?  If I didn't miss it, because there is nothing there, could
you please follow up to Joe's earlier question?

Thanks,

Sander

[mturk]

Sander Striker wrote:

> On Wed, Apr 16, 2008 at 3:52 PM, Joe Orton <jorton@...> wrote:
>> On Tue, Apr 15, 2008 at 06:48:49AM +0200, Mladen Turk wrote:
>>> Joe Orton wrote:
>>>> On Sat, Apr 12, 2008 at 08:42:59AM -0000, Mladen Turk wrote:
>>>>> Author: mturk
>>>>> Date: Sat Apr 12 01:42:51 2008
>>>>> New Revision: 647394
>>>>>
>>>>> URL: http://svn.apache.org/viewvc?rev=647394&view=rev
>>>>> Log:
>>>>> Introduce apr_pool_sys_allocator_set
>> Mladen - I think you posted that response to this thread as well as the
>> other; this thread was about r647384 (apr_pool_create_core_ex) not
>> r647394 (apr_pool_sys_allocator_set).
>
> I checked on the archives, as I stumbled over the code and wondered
> what the heck it was for.  I don't seem to find a response as to what
> this functionality is actually needed for.  In case I missed it, could you
> point me to it?  If I didn't miss it, because there is nothing there, could
> you please follow up to Joe's earlier question?
>

Sorry for the late answer. I was on the vacation ;)

I think I already explained the purpose and usage of create_core_ex.
Unlike traditional pool it creates standalone pool (and allocator if
provided at creation time).
The usage addresses two main common concepts:
1. Temporary memory pools that are commonly used inside loops
    and are used as create/alloc*n/destroy
    Since create_core_ex doesn't lock the parent (or global) pool
    during create/destroy this is performance upgrade.
2. Using pools with high level languages like Java that have
    strict garbage collection. APR Objects in that case can be
    directly bound to the java object without the need to sync the
    GC with the pool maintenance. Since those languages ensure that
    the object destroy method will be called, but in random order
    there is no need to create any back reference to parent pool.

Just to give you an overview of the problems with that in heavily
multithreaded environment like JVM where GC and occur in parallel,
are the long JNI calls in which case the apr_terminate needs to be
protected by the sync method so that it gets called *only* when all
functions exits. In any other case we might end up with zombie
memory (child pool) in the middle of the function call.
We have that problem in Tomcat Native, so a simple high CPU usage
can lead to a JVM core during connector shutdown/restart. JVM can
decide to cleanup the our apr socket object at some later time when
the parent was already destroyed. To solve this issue we would need
to create internal tree that would map apr_pool parent/child dependency
1:1, protect the access and maintain it, as well as having an atomic
counter of number of JNI functions currently executing.
With detached pools we don't need that and the memory will still
be released by calling pool_destroy in objects finalize method.

This breaks the single root pool presumption, so the API has to be used
only in well defined environments where for each create_core_ex the
corresponding destroy will be always called.


Long one, I know :)

Regards
--
^(TM)