swig + java + gc

[lytles]

i'm using swig to wrap native lapack implementations (mkl and acml).
i've fixed several issues (signal chaining and premature collection)
and have a pretty good system. but i'm struggling to manage the
memory.

to test, i'm looping over a test suite. each iteration allocates and
sets memory (using cpointer.i and carrays.i) and makes some lapack
calls. no references to previous iterations are preserved.

if i call System.gc() after each iteration, the vmsize of the app
never seems to grow. but if i don't call the gc(), the vmsize grows
without bound and appears to crash the machine (not sure why the
oom-killer doesn't kill it). and if i call System.gc() every once in a
while, the size grows more slowly, but may or may not grow without
bound. and once swig memory has been around for a while (eg hasn't
been reclaimed in it's first 60 seconds), it will never be reclaimed.

it seems that the effectiveness of gc() at reclaiming the swig
resources is dependent on how much java memory allocation has been
done relative to the swig / native allocation. i've tried a number of
work-arounds.

- "instrument" the user code. call gc() frequently in the same thread
as the user code. this works fine and by "timing" it right i can
achieve very slow vmsize growth without slowing things down much. but
this needs to be transparent to the user (i'm writing a library for a
scripting environment), so it means i'd need to modify many of the
entry or exit points and make sure the gc() was run if needed. this
solution is effective but messy, and just seems wrong.

- call gc() from another thread. i've tried this before and found
operating points at which it broke down, but right now it seems work
ok. vmsize grows but seems to stabilize.

- allocate java memory and call gc() from another thread. more
consistent but slower than just calling gc(). sensitive to java max
heap size.

i'm seeing a few crashes with the latter two techniques unless i use
locking (in which case i might as well use the first method). i assume
that this is because i still need to clean up a bunch of premature
collections. i'm also using openmp (well, mkl and acml are) - not sure
if this conflicts with using multiple threads in java (sure hope not).
if i'm still confused after doing some investigation, i'll post a
separate note about openmp.


wondering if others have seen similar behavior (or it's absence) when
allocating lots of memory on the swig side of the jni. i'm doing 10k -
50 meg per iteration, and very little non-jni allocation.



notes:
two test machines:
  intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
  amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
measuring vmsize using "cat /proc/stat" and using the vmsize entry
interface files are attached




-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[Lubomir Konstantinov-2]

Hi,

I'm using swig in market data applications where the amount of data
flowing thru is, well...huge. The memory stays nice and tidy as long
as I make sure to dispose of the objects myself, without relying on
the gc to clean them up.

Cheers,
Lubo

On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <seth.lytle@...> wrote:

> i'm using swig to wrap native lapack implementations (mkl and acml).
> i've fixed several issues (signal chaining and premature collection)
> and have a pretty good system. but i'm struggling to manage the
> memory.
>
> to test, i'm looping over a test suite. each iteration allocates and
> sets memory (using cpointer.i and carrays.i) and makes some lapack
> calls. no references to previous iterations are preserved.
>
> if i call System.gc() after each iteration, the vmsize of the app
> never seems to grow. but if i don't call the gc(), the vmsize grows
> without bound and appears to crash the machine (not sure why the
> oom-killer doesn't kill it). and if i call System.gc() every once in a
> while, the size grows more slowly, but may or may not grow without
> bound. and once swig memory has been around for a while (eg hasn't
> been reclaimed in it's first 60 seconds), it will never be reclaimed.
>
> it seems that the effectiveness of gc() at reclaiming the swig
> resources is dependent on how much java memory allocation has been
> done relative to the swig / native allocation. i've tried a number of
> work-arounds.
>
> - "instrument" the user code. call gc() frequently in the same thread
> as the user code. this works fine and by "timing" it right i can
> achieve very slow vmsize growth without slowing things down much. but
> this needs to be transparent to the user (i'm writing a library for a
> scripting environment), so it means i'd need to modify many of the
> entry or exit points and make sure the gc() was run if needed. this
> solution is effective but messy, and just seems wrong.
>
> - call gc() from another thread. i've tried this before and found
> operating points at which it broke down, but right now it seems work
> ok. vmsize grows but seems to stabilize.
>
> - allocate java memory and call gc() from another thread. more
> consistent but slower than just calling gc(). sensitive to java max
> heap size.
>
> i'm seeing a few crashes with the latter two techniques unless i use
> locking (in which case i might as well use the first method). i assume
> that this is because i still need to clean up a bunch of premature
> collections. i'm also using openmp (well, mkl and acml are) - not sure
> if this conflicts with using multiple threads in java (sure hope not).
> if i'm still confused after doing some investigation, i'll post a
> separate note about openmp.
>
>
> wondering if others have seen similar behavior (or it's absence) when
> allocating lots of memory on the swig side of the jni. i'm doing 10k -
> 50 meg per iteration, and very little non-jni allocation.
>
>
>
> notes:
> two test machines:
>  intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
>  amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
> measuring vmsize using "cat /proc/stat" and using the vmsize entry
> interface files are attached
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user
>
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[lytles]

i'm doing market data too. by "dispose of the objects myself" do you
mean thru the finalizer, or directly (eg, by calling
doubleArray.delete()) ? i don't think i have the option of doing it
directly, since the user code could still have a reference. assuming
that you're doing it directly, why did you decide to manage the memory
yourself instead of using the gc / finalizer ? did you run into
similar problems ?

seth


On Mon, Jun 9, 2008 at 2:55 PM, Lubomir Konstantinov <lubomir@...> wrote:

>
> Hi,
>
> I'm using swig in market data applications where the amount of data
> flowing thru is, well...huge. The memory stays nice and tidy as long
> as I make sure to dispose of the objects myself, without relying on
> the gc to clean them up.
>
> Cheers,
> Lubo
>
> On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <seth.lytle@...> wrote:
> > i'm using swig to wrap native lapack implementations (mkl and acml).
> > i've fixed several issues (signal chaining and premature collection)
> > and have a pretty good system. but i'm struggling to manage the
> > memory.
> >
> > to test, i'm looping over a test suite. each iteration allocates and
> > sets memory (using cpointer.i and carrays.i) and makes some lapack
> > calls. no references to previous iterations are preserved.
> >
> > if i call System.gc() after each iteration, the vmsize of the app
> > never seems to grow. but if i don't call the gc(), the vmsize grows
> > without bound and appears to crash the machine (not sure why the
> > oom-killer doesn't kill it). and if i call System.gc() every once in a
> > while, the size grows more slowly, but may or may not grow without
> > bound. and once swig memory has been around for a while (eg hasn't
> > been reclaimed in it's first 60 seconds), it will never be reclaimed.
> >
> > it seems that the effectiveness of gc() at reclaiming the swig
> > resources is dependent on how much java memory allocation has been
> > done relative to the swig / native allocation. i've tried a number of
> > work-arounds.
> >
> > - "instrument" the user code. call gc() frequently in the same thread
> > as the user code. this works fine and by "timing" it right i can
> > achieve very slow vmsize growth without slowing things down much. but
> > this needs to be transparent to the user (i'm writing a library for a
> > scripting environment), so it means i'd need to modify many of the
> > entry or exit points and make sure the gc() was run if needed. this
> > solution is effective but messy, and just seems wrong.
> >
> > - call gc() from another thread. i've tried this before and found
> > operating points at which it broke down, but right now it seems work
> > ok. vmsize grows but seems to stabilize.
> >
> > - allocate java memory and call gc() from another thread. more
> > consistent but slower than just calling gc(). sensitive to java max
> > heap size.
> >
> > i'm seeing a few crashes with the latter two techniques unless i use
> > locking (in which case i might as well use the first method). i assume
> > that this is because i still need to clean up a bunch of premature
> > collections. i'm also using openmp (well, mkl and acml are) - not sure
> > if this conflicts with using multiple threads in java (sure hope not).
> > if i'm still confused after doing some investigation, i'll post a
> > separate note about openmp.
> >
> >
> > wondering if others have seen similar behavior (or it's absence) when
> > allocating lots of memory on the swig side of the jni. i'm doing 10k -
> > 50 meg per iteration, and very little non-jni allocation.
> >
> >
> >
> > notes:
> > two test machines:
> >  intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
> >  amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
> > measuring vmsize using "cat /proc/stat" and using the vmsize entry
> > interface files are attached
> >
> > -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://sourceforge.net/services/buy/index.php
> > _______________________________________________
> > Swig-user mailing list
> > Swig-user@...
> > https://lists.sourceforge.net/lists/listinfo/swig-user
> >
> >

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[Brian Cole-3]

We ran into this problem last week. The best solution we have found so  
far is tuning the garbage collector to work harder using the `-
XX:GCTimeRatio=<nnn>` option (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.Ergonomics%20in%20the%20throughput%20collector%7Coutline 
). This with a combination of a better understanding from our users  
about memory issues seems to keep memory consumption in check.

However, this is a really undesirable solution for us as we distribute  
libraries to customers, and thus we don't have control over the  
command line options to the JVM. For the time being there doesn't seem  
to be another way around this problem. Our theory is that Java can  
create objects faster than the garbage collector can clean them up  
again. We think our next step is to see if speeding up the delete  
method will allow the garbage collector to keep pace with applications.

Is the problem rooted in the jni? Or swig's use of the jni? Or is it  
possible to overrun the JVM with straight java code?

-Brian

On Jun 9, 2008, at 1:41 PM, seth lytle wrote:

> i'm doing market data too. by "dispose of the objects myself" do you
> mean thru the finalizer, or directly (eg, by calling
> doubleArray.delete()) ? i don't think i have the option of doing it
> directly, since the user code could still have a reference. assuming
> that you're doing it directly, why did you decide to manage the memory
> yourself instead of using the gc / finalizer ? did you run into
> similar problems ?
>
> seth
>
>
> On Mon, Jun 9, 2008 at 2:55 PM, Lubomir Konstantinov <lubomir@...
> > wrote:
>>
>> Hi,
>>
>> I'm using swig in market data applications where the amount of data
>> flowing thru is, well...huge. The memory stays nice and tidy as long
>> as I make sure to dispose of the objects myself, without relying on
>> the gc to clean them up.
>>
>> Cheers,
>> Lubo
>>
>> On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <seth.lytle@...>  
>> wrote:
>>> i'm using swig to wrap native lapack implementations (mkl and acml).
>>> i've fixed several issues (signal chaining and premature collection)
>>> and have a pretty good system. but i'm struggling to manage the
>>> memory.
>>>
>>> to test, i'm looping over a test suite. each iteration allocates and
>>> sets memory (using cpointer.i and carrays.i) and makes some lapack
>>> calls. no references to previous iterations are preserved.
>>>
>>> if i call System.gc() after each iteration, the vmsize of the app
>>> never seems to grow. but if i don't call the gc(), the vmsize grows
>>> without bound and appears to crash the machine (not sure why the
>>> oom-killer doesn't kill it). and if i call System.gc() every once  
>>> in a
>>> while, the size grows more slowly, but may or may not grow without
>>> bound. and once swig memory has been around for a while (eg hasn't
>>> been reclaimed in it's first 60 seconds), it will never be  
>>> reclaimed.
>>>
>>> it seems that the effectiveness of gc() at reclaiming the swig
>>> resources is dependent on how much java memory allocation has been
>>> done relative to the swig / native allocation. i've tried a number  
>>> of
>>> work-arounds.
>>>
>>> - "instrument" the user code. call gc() frequently in the same  
>>> thread
>>> as the user code. this works fine and by "timing" it right i can
>>> achieve very slow vmsize growth without slowing things down much.  
>>> but
>>> this needs to be transparent to the user (i'm writing a library  
>>> for a
>>> scripting environment), so it means i'd need to modify many of the
>>> entry or exit points and make sure the gc() was run if needed. this
>>> solution is effective but messy, and just seems wrong.
>>>
>>> - call gc() from another thread. i've tried this before and found
>>> operating points at which it broke down, but right now it seems work
>>> ok. vmsize grows but seems to stabilize.
>>>
>>> - allocate java memory and call gc() from another thread. more
>>> consistent but slower than just calling gc(). sensitive to java max
>>> heap size.
>>>
>>> i'm seeing a few crashes with the latter two techniques unless i use
>>> locking (in which case i might as well use the first method). i  
>>> assume
>>> that this is because i still need to clean up a bunch of premature
>>> collections. i'm also using openmp (well, mkl and acml are) - not  
>>> sure
>>> if this conflicts with using multiple threads in java (sure hope  
>>> not).
>>> if i'm still confused after doing some investigation, i'll post a
>>> separate note about openmp.
>>>
>>>
>>> wondering if others have seen similar behavior (or it's absence)  
>>> when
>>> allocating lots of memory on the swig side of the jni. i'm doing  
>>> 10k -
>>> 50 meg per iteration, and very little non-jni allocation.
>>>
>>>
>>>
>>> notes:
>>> two test machines:
>>> intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
>>> amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
>>> measuring vmsize using "cat /proc/stat" and using the vmsize entry
>>> interface files are attached
>>>
>>> -------------------------------------------------------------------------
>>> Check out the new SourceForge.net Marketplace.
>>> It's the best place to buy or sell services for
>>> just about anything Open Source.
>>> http://sourceforge.net/services/buy/index.php
>>> _______________________________________________
>>> Swig-user mailing list
>>> Swig-user@...
>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>>
>>>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[Lubomir Konstantinov-2]

IMHO, its not a problem with swig, but with the way GC works.

 I've successfuly dealt with a similiar problem by delete()-ing the
objects myself + tuning the GC, so it keeps them in the young
generation and stops them from overflowing to the tenured one. AFAIK,
you cannot override the GC behaviour from Java code.

On Mon, Jun 9, 2008 at 11:56 PM, Brian Cole <coleb@...> wrote:

> We ran into this problem last week. The best solution we have found so
> far is tuning the garbage collector to work harder using the `-
> XX:GCTimeRatio=<nnn>` option (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.Ergonomics%20in%20the%20throughput%20collector%7Coutline
> ). This with a combination of a better understanding from our users
> about memory issues seems to keep memory consumption in check.
>
> However, this is a really undesirable solution for us as we distribute
> libraries to customers, and thus we don't have control over the
> command line options to the JVM. For the time being there doesn't seem
> to be another way around this problem. Our theory is that Java can
> create objects faster than the garbage collector can clean them up
> again. We think our next step is to see if speeding up the delete
> method will allow the garbage collector to keep pace with applications.
>
> Is the problem rooted in the jni? Or swig's use of the jni? Or is it
> possible to overrun the JVM with straight java code?
>
> -Brian
>
> On Jun 9, 2008, at 1:41 PM, seth lytle wrote:
>
>> i'm doing market data too. by "dispose of the objects myself" do you
>> mean thru the finalizer, or directly (eg, by calling
>> doubleArray.delete()) ? i don't think i have the option of doing it
>> directly, since the user code could still have a reference. assuming
>> that you're doing it directly, why did you decide to manage the memory
>> yourself instead of using the gc / finalizer ? did you run into
>> similar problems ?
>>
>> seth
>>
>>
>> On Mon, Jun 9, 2008 at 2:55 PM, Lubomir Konstantinov <lubomir@...
>> > wrote:
>>>
>>> Hi,
>>>
>>> I'm using swig in market data applications where the amount of data
>>> flowing thru is, well...huge. The memory stays nice and tidy as long
>>> as I make sure to dispose of the objects myself, without relying on
>>> the gc to clean them up.
>>>
>>> Cheers,
>>> Lubo
>>>
>>> On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <seth.lytle@...>
>>> wrote:
>>>> i'm using swig to wrap native lapack implementations (mkl and acml).
>>>> i've fixed several issues (signal chaining and premature collection)
>>>> and have a pretty good system. but i'm struggling to manage the
>>>> memory.
>>>>
>>>> to test, i'm looping over a test suite. each iteration allocates and
>>>> sets memory (using cpointer.i and carrays.i) and makes some lapack
>>>> calls. no references to previous iterations are preserved.
>>>>
>>>> if i call System.gc() after each iteration, the vmsize of the app
>>>> never seems to grow. but if i don't call the gc(), the vmsize grows
>>>> without bound and appears to crash the machine (not sure why the
>>>> oom-killer doesn't kill it). and if i call System.gc() every once
>>>> in a
>>>> while, the size grows more slowly, but may or may not grow without
>>>> bound. and once swig memory has been around for a while (eg hasn't
>>>> been reclaimed in it's first 60 seconds), it will never be
>>>> reclaimed.
>>>>
>>>> it seems that the effectiveness of gc() at reclaiming the swig
>>>> resources is dependent on how much java memory allocation has been
>>>> done relative to the swig / native allocation. i've tried a number
>>>> of
>>>> work-arounds.
>>>>
>>>> - "instrument" the user code. call gc() frequently in the same
>>>> thread
>>>> as the user code. this works fine and by "timing" it right i can
>>>> achieve very slow vmsize growth without slowing things down much.
>>>> but
>>>> this needs to be transparent to the user (i'm writing a library
>>>> for a
>>>> scripting environment), so it means i'd need to modify many of the
>>>> entry or exit points and make sure the gc() was run if needed. this
>>>> solution is effective but messy, and just seems wrong.
>>>>
>>>> - call gc() from another thread. i've tried this before and found
>>>> operating points at which it broke down, but right now it seems work
>>>> ok. vmsize grows but seems to stabilize.
>>>>
>>>> - allocate java memory and call gc() from another thread. more
>>>> consistent but slower than just calling gc(). sensitive to java max
>>>> heap size.
>>>>
>>>> i'm seeing a few crashes with the latter two techniques unless i use
>>>> locking (in which case i might as well use the first method). i
>>>> assume
>>>> that this is because i still need to clean up a bunch of premature
>>>> collections. i'm also using openmp (well, mkl and acml are) - not
>>>> sure
>>>> if this conflicts with using multiple threads in java (sure hope
>>>> not).
>>>> if i'm still confused after doing some investigation, i'll post a
>>>> separate note about openmp.
>>>>
>>>>
>>>> wondering if others have seen similar behavior (or it's absence)
>>>> when
>>>> allocating lots of memory on the swig side of the jni. i'm doing
>>>> 10k -
>>>> 50 meg per iteration, and very little non-jni allocation.
>>>>
>>>>
>>>>
>>>> notes:
>>>> two test machines:
>>>> intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
>>>> amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
>>>> measuring vmsize using "cat /proc/stat" and using the vmsize entry
>>>> interface files are attached
>>>>
>>>> -------------------------------------------------------------------------
>>>> Check out the new SourceForge.net Marketplace.
>>>> It's the best place to buy or sell services for
>>>> just about anything Open Source.
>>>> http://sourceforge.net/services/buy/index.php
>>>> _______________________________________________
>>>> Swig-user mailing list
>>>> Swig-user@...
>>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>
>>>>
>>
>> -------------------------------------------------------------------------
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services for
>> just about anything Open Source.
>> http://sourceforge.net/services/buy/index.php
>> _______________________________________________
>> Swig-user mailing list
>> Swig-user@...
>> https://lists.sourceforge.net/lists/listinfo/swig-user
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[wsfulton]

Agreed it isn't necessarily a SWIG problem, it is a consequence of the
commonly used 'peer' or 'proxy' design pattern used with JNI code where
a finalizer has to be called to clear up the underlying memory. GC
tuning is the most common and 'best practice' solution. Also make sure
that the finalizer thread has time to run. If not, the gc might run and
mark SWIG objects for finalization but they never get finalized. I've
found that the garbage collector will even give up and run out of memory
even with objects in the finalization queue. That is quite easy to test
by creating a never ending loop that simply creates SWIG objects that
never get used. A 'Sleep' within the loop solves the out of memory problem.

There is also a novel approach in the documentation you might want to
have a look at:

http://www.swig.org/Doc1.3/Java.html#java_heap_allocations. I found it
created a performance hit though.

William

Lubomir Konstantinov wrote:

> IMHO, its not a problem with swig, but with the way GC works.
>
>  I've successfuly dealt with a similiar problem by delete()-ing the
> objects myself + tuning the GC, so it keeps them in the young
> generation and stops them from overflowing to the tenured one. AFAIK,
> you cannot override the GC behaviour from Java code.
>
> On Mon, Jun 9, 2008 at 11:56 PM, Brian Cole <coleb@...> wrote:
>> We ran into this problem last week. The best solution we have found so
>> far is tuning the garbage collector to work harder using the `-
>> XX:GCTimeRatio=<nnn>` option (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.Ergonomics%20in%20the%20throughput%20collector%7Coutline
>> ). This with a combination of a better understanding from our users
>> about memory issues seems to keep memory consumption in check.
>>
>> However, this is a really undesirable solution for us as we distribute
>> libraries to customers, and thus we don't have control over the
>> command line options to the JVM. For the time being there doesn't seem
>> to be another way around this problem. Our theory is that Java can
>> create objects faster than the garbage collector can clean them up
>> again. We think our next step is to see if speeding up the delete
>> method will allow the garbage collector to keep pace with applications.
>>
>> Is the problem rooted in the jni? Or swig's use of the jni? Or is it
>> possible to overrun the JVM with straight java code?
>>
>> -Brian
>>
>> On Jun 9, 2008, at 1:41 PM, seth lytle wrote:
>>
>>> i'm doing market data too. by "dispose of the objects myself" do you
>>> mean thru the finalizer, or directly (eg, by calling
>>> doubleArray.delete()) ? i don't think i have the option of doing it
>>> directly, since the user code could still have a reference. assuming
>>> that you're doing it directly, why did you decide to manage the memory
>>> yourself instead of using the gc / finalizer ? did you run into
>>> similar problems ?
>>>
>>> seth
>>>
>>>
>>> On Mon, Jun 9, 2008 at 2:55 PM, Lubomir Konstantinov <lubomir@...
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm using swig in market data applications where the amount of data
>>>> flowing thru is, well...huge. The memory stays nice and tidy as long
>>>> as I make sure to dispose of the objects myself, without relying on
>>>> the gc to clean them up.
>>>>
>>>> Cheers,
>>>> Lubo
>>>>
>>>> On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <seth.lytle@...>
>>>> wrote:
>>>>> i'm using swig to wrap native lapack implementations (mkl and acml).
>>>>> i've fixed several issues (signal chaining and premature collection)
>>>>> and have a pretty good system. but i'm struggling to manage the
>>>>> memory.
>>>>>
>>>>> to test, i'm looping over a test suite. each iteration allocates and
>>>>> sets memory (using cpointer.i and carrays.i) and makes some lapack
>>>>> calls. no references to previous iterations are preserved.
>>>>>
>>>>> if i call System.gc() after each iteration, the vmsize of the app
>>>>> never seems to grow. but if i don't call the gc(), the vmsize grows
>>>>> without bound and appears to crash the machine (not sure why the
>>>>> oom-killer doesn't kill it). and if i call System.gc() every once
>>>>> in a
>>>>> while, the size grows more slowly, but may or may not grow without
>>>>> bound. and once swig memory has been around for a while (eg hasn't
>>>>> been reclaimed in it's first 60 seconds), it will never be
>>>>> reclaimed.
>>>>>
>>>>> it seems that the effectiveness of gc() at reclaiming the swig
>>>>> resources is dependent on how much java memory allocation has been
>>>>> done relative to the swig / native allocation. i've tried a number
>>>>> of
>>>>> work-arounds.
>>>>>
>>>>> - "instrument" the user code. call gc() frequently in the same
>>>>> thread
>>>>> as the user code. this works fine and by "timing" it right i can
>>>>> achieve very slow vmsize growth without slowing things down much.
>>>>> but
>>>>> this needs to be transparent to the user (i'm writing a library
>>>>> for a
>>>>> scripting environment), so it means i'd need to modify many of the
>>>>> entry or exit points and make sure the gc() was run if needed. this
>>>>> solution is effective but messy, and just seems wrong.
>>>>>
>>>>> - call gc() from another thread. i've tried this before and found
>>>>> operating points at which it broke down, but right now it seems work
>>>>> ok. vmsize grows but seems to stabilize.
>>>>>
>>>>> - allocate java memory and call gc() from another thread. more
>>>>> consistent but slower than just calling gc(). sensitive to java max
>>>>> heap size.
>>>>>
>>>>> i'm seeing a few crashes with the latter two techniques unless i use
>>>>> locking (in which case i might as well use the first method). i
>>>>> assume
>>>>> that this is because i still need to clean up a bunch of premature
>>>>> collections. i'm also using openmp (well, mkl and acml are) - not
>>>>> sure
>>>>> if this conflicts with using multiple threads in java (sure hope
>>>>> not).
>>>>> if i'm still confused after doing some investigation, i'll post a
>>>>> separate note about openmp.
>>>>>
>>>>>
>>>>> wondering if others have seen similar behavior (or it's absence)
>>>>> when
>>>>> allocating lots of memory on the swig side of the jni. i'm doing
>>>>> 10k -
>>>>> 50 meg per iteration, and very little non-jni allocation.
>>>>>
>>>>>
>>>>>
>>>>> notes:
>>>>> two test machines:
>>>>> intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
>>>>> amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
>>>>> measuring vmsize using "cat /proc/stat" and using the vmsize entry
>>>>> interface files are attached
>>>>>
>>>>> -------------------------------------------------------------------------
>>>>> Check out the new SourceForge.net Marketplace.
>>>>> It's the best place to buy or sell services for
>>>>> just about anything Open Source.
>>>>> http://sourceforge.net/services/buy/index.php
>>>>> _______________________________________________
>>>>> Swig-user mailing list
>>>>> Swig-user@...
>>>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>>
>>>>>
>>> -------------------------------------------------------------------------
>>> Check out the new SourceForge.net Marketplace.
>>> It's the best place to buy or sell services for
>>> just about anything Open Source.
>>> http://sourceforge.net/services/buy/index.php
>>> _______________________________________________
>>> Swig-user mailing list
>>> Swig-user@...
>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>> -------------------------------------------------------------------------
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services for
>> just about anything Open Source.
>> http://sourceforge.net/services/buy/index.php
>> _______________________________________________
>> Swig-user mailing list
>> Swig-user@...
>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user