A question about last memory consumption patches

[José María Fernández González-2]

Hi everybody,
        I have seen that last Wolf's commits are related to reduce eXist nodesets
memory consumption, but they are being committed to eXist-stable-1.2 branch.
Are those changes already in trunk? I'm very interested on them due the
scenarios where I'm using eXist.

        Best Regards,
                José María
--
"There is no reason why anybody would want a computer in their home" -
        Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???

"Premature optimization is the root of all evil." - Donald Knuth

José María Fernández González
Tlfn: ([phone] / 91 224 69 00 (ext 3061)
e-mail: jmfernandez@... Fax: [phone]
Unidad del Instituto Nacional de Bioinformática
Biología Estructural y Biocomputación Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncológicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fernández Almagro, 3 Madrid (Spain)

**NOTA DE CONFIDENCIALIDAD** Este correo electrónico, y en su caso los ficheros adjuntos, pueden contener información protegida para el uso exclusivo de su destinatario. Se prohíbe la distribución, reproducción o cualquier otro tipo de transmisión por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Dannes Wessels]

Hi,

On Mon, Jun 23, 2008 at 9:24 PM, José María Fernández González
<jmfernandez@...> wrote:
>        I have seen that last Wolf's commits are related to reduce eXist nodesets
> memory consumption, but they are being committed to eXist-stable-1.2 branch.
> Are those changes already in trunk? I'm very interested on them due the
> scenarios where I'm using eXist.

Due to timing constraints wolfgang decided to improve/update 1.2.x
first. When time allows, the changes will be synced to trunk.

regards

Dannes

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Wolfgang Meier-2]

Hi José María,

>        I have seen that last Wolf's commits are related to reduce eXist nodesets
> memory consumption, but they are being committed to eXist-stable-1.2 branch.
> Are those changes already in trunk? I'm very interested on them due the
> scenarios where I'm using eXist.

I made those changes while profiling a set of concrete queries
generated by a complex application. To measure the effects and compare
the query times, it was easier and faster to directly work on the 1.2
branch. All changes will be ported to trunk soon.

The improvements in the 1.2 branch mainly apply to:

1) queries on a large amount of rather small documents
2) queries that generate a large number of in-memory XML fragments

in both cases, memory consumption has been reduced drastically, which
has a very positive effect on overall performance (some of my test
queries run several times faster now).

Wolfgang
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[José María Fernández González-2]

Hi Dannes & Wolf,
        thanks for your fast response!

        Best Regards,
                José María

Wolfgang Meier wrote:

> Hi José María,
>
>>        I have seen that last Wolf's commits are related to reduce eXist nodesets
>> memory consumption, but they are being committed to eXist-stable-1.2 branch.
>> Are those changes already in trunk? I'm very interested on them due the
>> scenarios where I'm using eXist.
>
> I made those changes while profiling a set of concrete queries
> generated by a complex application. To measure the effects and compare
> the query times, it was easier and faster to directly work on the 1.2
> branch. All changes will be ported to trunk soon.
>
> The improvements in the 1.2 branch mainly apply to:
>
> 1) queries on a large amount of rather small documents
> 2) queries that generate a large number of in-memory XML fragments
>
> in both cases, memory consumption has been reduced drastically, which
> has a very positive effect on overall performance (some of my test
> queries run several times faster now).
>
> Wolfgang

--
"There is no reason why anybody would want a computer in their home" -
        Ken Olson, founder of DEC 1977
"640K ought to be enough for anybody" - Bill Gates, 1981
"Nobody will ever outgrow a 20Mb hard drive." - ???

"Premature optimization is the root of all evil." - Donald Knuth

José María Fernández González
Tlfn: ([phone] / 91 224 69 00 (ext 3061)
e-mail: jmfernandez@... Fax: [phone]
Unidad del Instituto Nacional de Bioinformática
Biología Estructural y Biocomputación Structural Biology and Biocomputing
Centro Nacional de Investigaciones Oncológicas
C.P.: 28029 Zip Code: 28029
C/. Melchor Fernández Almagro, 3 Madrid (Spain)

**NOTA DE CONFIDENCIALIDAD** Este correo electr�nico, y en su caso los ficheros adjuntos, pueden contener informaci�n protegida para el uso exclusivo de su destinatario. Se proh�be la distribuci�n, reproducci�n o cualquier otro tipo de transmisi�n por parte de otra persona que no sea el destinatario. Si usted recibe por error este correo, se ruega comunicarlo al remitente y borrar el mensaje recibido.
**CONFIDENTIALITY NOTICE** This email communication and any attachments may contain confidential and privileged information for the sole use of the designated recipient named above. Distribution, reproduction or any other use of this transmission by any party other than the intended recipient is prohibited. If you are not the intended recipient please contact the sender and delete all copies.



-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Wolfgang Meier-2]

 > The improvements in the 1.2 branch mainly apply to:
 >
 > 1) queries on a large amount of rather small documents
 > 2) queries that generate a large number of in-memory XML fragments

I have now ported all performance-related commits from the 1.2 branch to
trunk. The main changes are:

1) the default implementation for node sets, ExtArrayNodeSet, has been
replaced by a more memory-efficient class: NewArrayNodeSet. The old
implementation was good for large documents, but consumed too much
memory per document. Since node sets are frequently created and passed
around during an average query, the memory overhead could be
considerable. The new implementation combines the advantages of the old
one (no changes to the core algorithms required) with a much smaller
memory footprint.

2) in memory fragments: some of the queries I profiled returned nothing
but a few thousand, separate XML fragments. The fragments were small
(just a few elements with attributes) and I was quite surprised about
the amount of memory they needed. Further analysis showed that most of
the memory was consumed by the document context attached to each
fragment, which mainly contained things like element and attribute names
or namespace declarations. This means wasting a lot of memory: most of
those data structures can be easily shared by all the document fragments
created within a query context (assuming that every thread has its own
context). I changed the implementation accordingly.

Due to the reduced memory usage and less overhead for garbage
collection, my test queries on a database with more than 100,000
documents got up to 5 times faster. We can't generalize this since my
queries were quite specific to a certain applicaton, but I would expect
to see a smaller improvement for completely different types of queries
as well.

Wolfgang

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Ben Bangert]

On Jun 25, 2008, at 3:02 PM, Wolfgang wrote:

>> The improvements in the 1.2 branch mainly apply to:
>>
>> 1) queries on a large amount of rather small documents
>> 2) queries that generate a large number of in-memory XML fragments

Is there going to be a 1.2.4 release with these improvements, or are  
there other bugs planned before the next release?

Cheers,
Ben

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Dannes Wessels]

Hi

On Thu, Jun 26, 2008 at 8:59 AM, Ben Bangert <ben@...> wrote:
> Is there going to be a 1.2.4 release with these improvements, or are there

yes, although there is no date set yet.

> other bugs planned before the next release?

no, we don;t plan bugs, we let you find them :-)

regards

Dannes

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open