Concurrency testing of the locationmap throws errors

[Thorsten Scherler-3]

Hi all,

I am ATM testing the dispatcher with jmeter and concurrent user.

The problem that I encounter is that the locationmap is throwing an
error as soon as there are concurrent user. :(

I have no clue ATM why that is happening but try
<map:match pattern="test">
 <map:generate src="lm://transform.html.document"/>
<map:serialize  />
</map:match>

with e.g. 10 user at the same time (Ramp-up time 0) and after you
freshly started forrest with skins or dispatcher. You will see 80%
failure with "Could not resolve locationmap".

I am quite doomed ATM since I had to finish my project yesterday and
after the last jmeter tests I cannot hand in the code.

Please if you have any idea what the problem can be let us now.

TIA

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Tim Williams]

On Jan 31, 2008 1:19 PM, Thorsten Scherler
<thorsten.scherler.ext@...> wrote:

> Hi all,
>
> I am ATM testing the dispatcher with jmeter and concurrent user.
>
> The problem that I encounter is that the locationmap is throwing an
> error as soon as there are concurrent user. :(
>
> I have no clue ATM why that is happening but try
> <map:match pattern="test">
>  <map:generate src="lm://transform.html.document"/>
> <map:serialize  />
> </map:match>
>
> with e.g. 10 user at the same time (Ramp-up time 0) and after you
> freshly started forrest with skins or dispatcher. You will see 80%
> failure with "Could not resolve locationmap".
>
> I am quite doomed ATM since I had to finish my project yesterday and
> after the last jmeter tests I cannot hand in the code.
>
> Please if you have any idea what the problem can be let us now.
>
> TIA

What are the errors? Stacktrace, etc?
--tim

[Thorsten Scherler-3]

On Thu, 2008-01-31 at 15:16 -0500, Tim Williams wrote:

> On Jan 31, 2008 1:19 PM, Thorsten Scherler
> <thorsten.scherler.ext@...> wrote:
> > Hi all,
> >
> > I am ATM testing the dispatcher with jmeter and concurrent user.
> >
> > The problem that I encounter is that the locationmap is throwing an
> > error as soon as there are concurrent user. :(
> >
> > I have no clue ATM why that is happening but try
> > <map:match pattern="test">
> >  <map:generate src="lm://transform.html.document"/>
> > <map:serialize  />
> > </map:match>
> >
> > with e.g. 10 user at the same time (Ramp-up time 0) and after you
> > freshly started forrest with skins or dispatcher. You will see 80%
> > failure with "Could not resolve locationmap".
> >
> > I am quite doomed ATM since I had to finish my project yesterday and
> > after the last jmeter tests I cannot hand in the code.
> >
> > Please if you have any idea what the problem can be let us now.
> >
> > TIA
>
> What are the errors? Stacktrace, etc?

2008-06-30 13:32:33.455::WARN:  /servicios.html
org.apache.cocoon.ProcessingException: Error during resolving of
'lm://resolve.structurer.servicios'.
        at <map:serialize type="xhtml"> -
file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:65:38
        at <map:transform> -
file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:64:70
        at <map:transform> -
file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:63:55
        at <map:transform type="dispatcher"> -
file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:56:42
....

Caused by: org.apache.excalibur.source.SourceException: Could not
resolve locationmap location.
        at
org.apache.forrest.locationmap.source.impl.LocationmapSourceFactory.getSource(LocationmapSourceFactory.java:81)
        at
org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
        at
org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
        at
org.apache.cocoon.template.script.DefaultScriptManager.resolveTemplate(DefaultScriptManager.java:97)


Doing some testing in my case I use jmeter with 250 threads and 100 loop
where the ramp-up time is 20.

This provokes that the input module is requested on the startup with
more then one request. It seems that the requests are issued as fast
that the module seems to fail to finish to load the first time the
locationmap.

If I however request FIRST (before the jmeter test) a page and let the
locationmap module load with time the tests will not fail anymore.

I am ATM playing around and it happens in
LocationMapModule.getLocationMap() but I am not sure how I can prevent
it.

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Tim Williams]

On Mon, Jun 30, 2008 at 7:46 AM, Thorsten Scherler
<thorsten.scherler.ext@...> wrote:

> On Thu, 2008-01-31 at 15:16 -0500, Tim Williams wrote:
>> On Jan 31, 2008 1:19 PM, Thorsten Scherler
>> <thorsten.scherler.ext@...> wrote:
>> > Hi all,
>> >
>> > I am ATM testing the dispatcher with jmeter and concurrent user.
>> >
>> > The problem that I encounter is that the locationmap is throwing an
>> > error as soon as there are concurrent user. :(
>> >
>> > I have no clue ATM why that is happening but try
>> > <map:match pattern="test">
>> >  <map:generate src="lm://transform.html.document"/>
>> > <map:serialize  />
>> > </map:match>
>> >
>> > with e.g. 10 user at the same time (Ramp-up time 0) and after you
>> > freshly started forrest with skins or dispatcher. You will see 80%
>> > failure with "Could not resolve locationmap".
>> >
>> > I am quite doomed ATM since I had to finish my project yesterday and
>> > after the last jmeter tests I cannot hand in the code.
>> >
>> > Please if you have any idea what the problem can be let us now.
>> >
>> > TIA
>>
>> What are the errors? Stacktrace, etc?
>
> 2008-06-30 13:32:33.455::WARN:  /servicios.html
> org.apache.cocoon.ProcessingException: Error during resolving of
> 'lm://resolve.structurer.servicios'.
>        at <map:serialize type="xhtml"> -
> file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:65:38
>        at <map:transform> -
> file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:64:70
>        at <map:transform> -
> file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:63:55
>        at <map:transform type="dispatcher"> -
> file:///home/thorsten/src/sadesi/portalv4/trunk/portal/target/work/blocks/dispatcher/sitemap.xmap:56:42
> ....
>
> Caused by: org.apache.excalibur.source.SourceException: Could not
> resolve locationmap location.
>        at
> org.apache.forrest.locationmap.source.impl.LocationmapSourceFactory.getSource(LocationmapSourceFactory.java:81)
>        at
> org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
>        at
> org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
>        at
> org.apache.cocoon.template.script.DefaultScriptManager.resolveTemplate(DefaultScriptManager.java:97)
>
>
> Doing some testing in my case I use jmeter with 250 threads and 100 loop
> where the ramp-up time is 20.
>
> This provokes that the input module is requested on the startup with
> more then one request. It seems that the requests are issued as fast
> that the module seems to fail to finish to load the first time the
> locationmap.
>
> If I however request FIRST (before the jmeter test) a page and let the
> locationmap module load with time the tests will not fail anymore.
>
> I am ATM playing around and it happens in
> LocationMapModule.getLocationMap() but I am not sure how I can prevent
> it.

Try making the entire method synchronized instead of just the blocks.
This may not be the end solution but will let you quickly see if
there's concurrency problems when the subsequent requests come in
before the LM is fully built.  Just my top of the head thought, sorry,
I don't have time to look further right now.
--tim

[Thorsten Scherler-3]

On Mon, 2008-06-30 at 08:05 -0400, Tim Williams wrote:
...
> > I am ATM playing around and it happens in
> > LocationMapModule.getLocationMap() but I am not sure how I can prevent
> > it.
>
> Try making the entire method synchronized instead of just the blocks.
> This may not be the end solution but will let you quickly see if
> there's concurrency problems when the subsequent requests come in
> before the LM is fully built.  Just my top of the head thought, sorry,
> I don't have time to look further right now.

Yeah, meanwhile I did that for the LocationMapModule
 private synchronized LocationMap getLocationMap() throws Exception {
...

and as well for LocationMap.build(final Configuration configuration)
 public synchronized void build

Now "just" the first lookup fails, meaning it cured quite a bit but not
fully yet.

Keeping you informed and thanks for your feedback tim.
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Tim Williams]

On Mon, Jun 30, 2008 at 8:12 AM, Thorsten Scherler
<thorsten.scherler.ext@...> wrote:

> On Mon, 2008-06-30 at 08:05 -0400, Tim Williams wrote:
> ...
>> > I am ATM playing around and it happens in
>> > LocationMapModule.getLocationMap() but I am not sure how I can prevent
>> > it.
>>
>> Try making the entire method synchronized instead of just the blocks.
>> This may not be the end solution but will let you quickly see if
>> there's concurrency problems when the subsequent requests come in
>> before the LM is fully built.  Just my top of the head thought, sorry,
>> I don't have time to look further right now.
>
> Yeah, meanwhile I did that for the LocationMapModule
>  private synchronized LocationMap getLocationMap() throws Exception {
> ...
>
> and as well for LocationMap.build(final Configuration configuration)
>  public synchronized void build
>
> Now "just" the first lookup fails, meaning it cured quite a bit but not
> fully yet.
>
> Keeping you informed and thanks for your feedback tim.

I reckon in that case, it's either the caching or its deeper in the
actual LM/nodes.  You could try turning off caching completely and see
if it helps.  Or, you could change the implementation from a HashMap
to a ConcurrentHashMap and see if that helps.

--tim

[Thorsten Scherler-3]

On Mon, 2008-06-30 at 08:34 -0400, Tim Williams wrote:
> On Mon, Jun 30, 2008 at 8:12 AM, Thorsten Scherler
...
>
> I reckon in that case, it's either the caching or its deeper in the
> actual LM/nodes.  You could try turning off caching completely and see
> if it helps.  

Turning caching off does not change the situation.

> Or, you could change the implementation from a HashMap
> to a ConcurrentHashMap and see if that helps.

I changed
public synchronized Object getAttribute(...)
which seems to fix the problem.

:)

Will commit now.

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Tim Williams]

On Mon, Jun 30, 2008 at 9:04 AM, Thorsten Scherler
<thorsten.scherler.ext@...> wrote:

> On Mon, 2008-06-30 at 08:34 -0400, Tim Williams wrote:
>> On Mon, Jun 30, 2008 at 8:12 AM, Thorsten Scherler
> ...
>>
>> I reckon in that case, it's either the caching or its deeper in the
>> actual LM/nodes.  You could try turning off caching completely and see
>> if it helps.
>
> Turning caching off does not change the situation.
>
>> Or, you could change the implementation from a HashMap
>> to a ConcurrentHashMap and see if that helps.
>
> I changed
> public synchronized Object getAttribute(...)
> which seems to fix the problem.
>
> :)

I'm not sure that having getAttribute synchronized is good long term.
Have you noticed a performance problems now?  Maybe create an issue to
review this and figure out what the real culprit is in there?

--tim

[Thorsten Scherler-3]

On Mon, 2008-06-30 at 09:14 -0400, Tim Williams wrote:

> On Mon, Jun 30, 2008 at 9:04 AM, Thorsten Scherler
> <thorsten.scherler.ext@...> wrote:svnc
> > On Mon, 2008-06-30 at 08:34 -0400, Tim Williams wrote:
> >> On Mon, Jun 30, 2008 at 8:12 AM, Thorsten Scherler
> > ...
> >>
> >> I reckon in that case, it's either the caching or its deeper in the
> >> actual LM/nodes.  You could try turning off caching completely and see
> >> if it helps.
> >
> > Turning caching off does not change the situation.
> >
> >> Or, you could change the implementation from a HashMap
> >> to a ConcurrentHashMap and see if that helps.
> >
> > I changed
> > public synchronized Object getAttribute(...)
> > which seems to fix the problem.
> >
> > :)
>
> I'm not sure that having getAttribute synchronized is good long term.
> Have you noticed a performance problems now?  Maybe create an issue to
> review this and figure out what the real culprit is in there?
>

I will test the performance but in any case I think this problem needs
to fixed since concurrency bugs are really ugly to debug and having this
problem in a high traffic site is a complete showstopper.

I agree that there may is a better solution then having getAttribute
synchronized, I will ask on cocoon-dev.

I created FOR-1082 and added a note to the code.

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Thorsten Scherler-3]

On Tue, 2008-07-01 at 09:04 +0200, Thorsten Scherler wrote:
...
> I will test the performance
...

Actually with the same test as before the performance is better.

Mainly because before the first results are errors and ruin the average.
Since now we have 0% error the first hits are around 2.5/sec and raise
to 5.1/sec.

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions

[Thorsten Scherler-3]

On Tue, 2008-07-01 at 09:25 +0200, Thorsten Scherler wrote:

> On Tue, 2008-07-01 at 09:04 +0200, Thorsten Scherler wrote:
> ...
> > I will test the performance
> ...
>
> Actually with the same test as before the performance is better.
>
> Mainly because before the first results are errors and ruin the average.
> Since now we have 0% error the first hits are around 2.5/sec and raise
> to 5.1/sec.

BTW the site I am testing is http://juntadeandalucia.es/
(dispatcher/cocoon-2.2 based) and now with 250 threads in parallel
doing each 100 request to different pages of the app brings me with lm
caching enabled a throughput of 52,5 /sec (sample size 8855 requests).

salu2
--
Thorsten Scherler                                 thorsten.at.apache.org
Open Source Java                      consulting, training and solutions