Terracotta integration

[Ari Zilka]

Hello all,

I work for Terracotta on JVM-level clustering type stuff.  I recently  
spoke with a Wicket + TC user who went to production and uncovered a  
nasty surprise.  I wanted to propose to this community that we  
consider incorporating the changes he needed to make in order to  
cluster and scale well.

He--Richard Wilkinson--documented his changes on his blog here: http://www.richard-wilkinson.co.uk/2008/06/22/more-on-terracotta-and-wicket/

I look forward to discussing the possible ways forward.

Cheers,

--Ari

[Matej Knopp-2]

Hi,

the thing is a bit more complicated than just that :) I'm not sure how
Wicket terractotta integration works currently. Last time I checked it
was recommended to use HTTPSessionStore which itself is a big drawback
as the HTTPSessionStore has certain backbutton problems that can not
be resolved at least until 1.5.

As for DiskPageStore, Wicket itself contains support for session
replication that is however targetted to "regular" clusters assuming
that after each request changed session properties are serialized and
replicated to backing nodes (and preferably immediately deserialized -
explanation below).

DiskPageStore's purpose is to store serialized pages on disk in order
to allow access to previous page versions and also to conserve session
memory usage. So DiskPageStore serializes page during each request.
The serialization happens whether wicket runs on cluster on not.
However, when wicket is running on cluster, the already serialized
data is used during session replication, so the pages are not
serialized again. This is an important optimization that is resulting
in fact that there is no additional serialization penalty when running
wicket on (regular) cluster.

When target node receives the serialized session entries during
replication and deserializes it (it is recommended to configure the
container to deserialize session properties immediately) the page
itself is not really deserialized. It is only stored in target node's
diskpagestore, so there is no page deserialization happening and the
actual page data is not held in memory on target node.

This is roughly the idea of running Wicket application on a standard
replicated cluster. Now I'm not sure about TC, but IIRC the default
Wicket TC setup used HTTPSessionStore (which doesn't contain any
"special" support for session replication).

I think it would be very nice to have TC leverage the existing
replication support of Wicket's DiskPageStore acting more like a
simple session replication. WDYT?

Cheers,
-Matej

On Sat, Jun 28, 2008 at 10:31 AM, Ari Zilka <ari@...> wrote:

> Hello all,
>
> I work for Terracotta on JVM-level clustering type stuff.  I recently spoke
> with a Wicket + TC user who went to production and uncovered a nasty
> surprise.  I wanted to propose to this community that we consider
> incorporating the changes he needed to make in order to cluster and scale
> well.
>
> He--Richard Wilkinson--documented his changes on his blog here:
> http://www.richard-wilkinson.co.uk/2008/06/22/more-on-terracotta-and-wicket/
>
> I look forward to discussing the possible ways forward.
>
> Cheers,
>
> --Ari
>

[richardwilko]

The current terracotta integration does use the HTTPSessionStore, infact the integration module forces the HTTPSessionStore using the bytecode manipulation stuff it uses.

What i've found is that the key with any terracotta clustering is that you cannot cluster page objects as this causes too much garbage, instead you have to cluster a byte array of the serialised page object instead (this is what my new IPageMapEntry implementation does).

Would replacing the disk based store in the default session store with one that stores serialised pages in some sort of terracotta distributed map in memory work?  Although would that still leave the last accessed page as a page object in memory?

Cheers,

Richard

Matej Knopp-2 wrote:

Hi,

the thing is a bit more complicated than just that :) I'm not sure how
Wicket terractotta integration works currently. Last time I checked it
was recommended to use HTTPSessionStore which itself is a big drawback
as the HTTPSessionStore has certain backbutton problems that can not
be resolved at least until 1.5.

As for DiskPageStore, Wicket itself contains support for session
replication that is however targetted to "regular" clusters assuming
that after each request changed session properties are serialized and
replicated to backing nodes (and preferably immediately deserialized -
explanation below).

DiskPageStore's purpose is to store serialized pages on disk in order
to allow access to previous page versions and also to conserve session
memory usage. So DiskPageStore serializes page during each request.
The serialization happens whether wicket runs on cluster on not.
However, when wicket is running on cluster, the already serialized
data is used during session replication, so the pages are not
serialized again. This is an important optimization that is resulting
in fact that there is no additional serialization penalty when running
wicket on (regular) cluster.

When target node receives the serialized session entries during
replication and deserializes it (it is recommended to configure the
container to deserialize session properties immediately) the page
itself is not really deserialized. It is only stored in target node's
diskpagestore, so there is no page deserialization happening and the
actual page data is not held in memory on target node.

This is roughly the idea of running Wicket application on a standard
replicated cluster. Now I'm not sure about TC, but IIRC the default
Wicket TC setup used HTTPSessionStore (which doesn't contain any
"special" support for session replication).

I think it would be very nice to have TC leverage the existing
replication support of Wicket's DiskPageStore acting more like a
simple session replication. WDYT?

Cheers,
-Matej

On Sat, Jun 28, 2008 at 10:31 AM, Ari Zilka <[e-mail]> wrote:
> Hello all,
>
> I work for Terracotta on JVM-level clustering type stuff.  I recently spoke
> with a Wicket + TC user who went to production and uncovered a nasty
> surprise.  I wanted to propose to this community that we consider
> incorporating the changes he needed to make in order to cluster and scale
> well.
>
> He--Richard Wilkinson--documented his changes on his blog here:
> http://www.richard-wilkinson.co.uk/2008/06/22/more-on-terracotta-and-wicket/
>
> I look forward to discussing the possible ways forward.
>
> Cheers,
>
> --Ari
>

[Matej Knopp-2]

There is one thing to realize about Wicket pages serialization. If you
just simply serialize page instance that contains references to other
pages, those referenced will be part of serialized page.

e.g. you have page A that contains references to page B

So take the following scenario.
User is on page B, does something to it, page B gets serialized
then user goes to page A, that contains reference to page B

page A modifies property on page B and gets serialized. Note that page
B also gets serialized (it's referenced from page A), but not as
separate serialized page, but as a part of serialized PageA.

After this user returns to page B (using back button). Will the
modifications to Page B (that page A did)  be visible? They won't.
Because last instance of PageB was only serialized as part of Page A
serialization, thus it can't be retrieved independently.

What disk page store does is that it  intercepts the page
serialization, so when page A is serialized and wicket detects
reference to another page, it only puts a placeholder to the
serialization stream and serializes the pageB independently.

So simple Objects.toByteArray(page) will work, but only for simple
scenarious without inter-page references.

What might be worth thinking of would be a terracotta based page store
(extending AbstractPageStore - class that implements the
(de)serialization magic). But even so it would still be necessary to
cluster http session.

-Matej

On Mon, Jun 30, 2008 at 11:00 AM, richardwilko
<richardjohnwilkinson@...> wrote:

>
> The current terracotta integration does use the HTTPSessionStore, infact the
> integration module forces the HTTPSessionStore using the bytecode
> manipulation stuff it uses.
>
> What i've found is that the key with any terracotta clustering is that you
> cannot cluster page objects as this causes too much garbage, instead you
> have to cluster a byte array of the serialised page object instead (this is
> what my new IPageMapEntry implementation does).
>
> Would replacing the disk based store in the default session store with one
> that stores serialised pages in some sort of terracotta distributed map in
> memory work?  Although would that still leave the last accessed page as a
> page object in memory?
>
> Cheers,
>
> Richard
>
>
>
> Matej Knopp-2 wrote:
>>
>> Hi,
>>
>> the thing is a bit more complicated than just that :) I'm not sure how
>> Wicket terractotta integration works currently. Last time I checked it
>> was recommended to use HTTPSessionStore which itself is a big drawback
>> as the HTTPSessionStore has certain backbutton problems that can not
>> be resolved at least until 1.5.
>>
>> As for DiskPageStore, Wicket itself contains support for session
>> replication that is however targetted to "regular" clusters assuming
>> that after each request changed session properties are serialized and
>> replicated to backing nodes (and preferably immediately deserialized -
>> explanation below).
>>
>> DiskPageStore's purpose is to store serialized pages on disk in order
>> to allow access to previous page versions and also to conserve session
>> memory usage. So DiskPageStore serializes page during each request.
>> The serialization happens whether wicket runs on cluster on not.
>> However, when wicket is running on cluster, the already serialized
>> data is used during session replication, so the pages are not
>> serialized again. This is an important optimization that is resulting
>> in fact that there is no additional serialization penalty when running
>> wicket on (regular) cluster.
>>
>> When target node receives the serialized session entries during
>> replication and deserializes it (it is recommended to configure the
>> container to deserialize session properties immediately) the page
>> itself is not really deserialized. It is only stored in target node's
>> diskpagestore, so there is no page deserialization happening and the
>> actual page data is not held in memory on target node.
>>
>> This is roughly the idea of running Wicket application on a standard
>> replicated cluster. Now I'm not sure about TC, but IIRC the default
>> Wicket TC setup used HTTPSessionStore (which doesn't contain any
>> "special" support for session replication).
>>
>> I think it would be very nice to have TC leverage the existing
>> replication support of Wicket's DiskPageStore acting more like a
>> simple session replication. WDYT?
>>
>> Cheers,
>> -Matej
>>
>> On Sat, Jun 28, 2008 at 10:31 AM, Ari Zilka <ari@...>
>> wrote:
>>> Hello all,
>>>
>>> I work for Terracotta on JVM-level clustering type stuff.  I recently
>>> spoke
>>> with a Wicket + TC user who went to production and uncovered a nasty
>>> surprise.  I wanted to propose to this community that we consider
>>> incorporating the changes he needed to make in order to cluster and scale
>>> well.
>>>
>>> He--Richard Wilkinson--documented his changes on his blog here:
>>> http://www.richard-wilkinson.co.uk/2008/06/22/more-on-terracotta-and-wicket/
>>>
>>> I look forward to discussing the possible ways forward.
>>>
>>> Cheers,
>>>
>>> --Ari
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18191341.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>
>

[richardwilko]

Matej Knopp-2 wrote:

What might be worth thinking of would be a terracotta based page store
(extending AbstractPageStore - class that implements the
(de)serialization magic). But even so it would still be necessary to
cluster http session.

Thats what kind of what I was thinking, and if we store this TerracottaPageStore in the httpsession, rather than on disk a la DiskPageStore, then we can just use terracotta to distribute the httpsession as normal and everything *should* work

[Ari Zilka]

that makes sense to me too.

--Ari

On Jun 30, 2008, at 3:05 AM, richardwilko wrote:

>
>
>
> Matej Knopp-2 wrote:
>>
>> What might be worth thinking of would be a terracotta based page  
>> store
>> (extending AbstractPageStore - class that implements the
>> (de)serialization magic). But even so it would still be necessary to
>> cluster http session.
>>
>
> Thats what kind of what I was thinking, and if we store this
> TerracottaPageStore in the httpsession, rather than on disk a la
> DiskPageStore, then we can just use terracotta to distribute the  
> httpsession
> as normal and everything *should* work
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18192381.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>

[Ari Zilka]

Ok,

a more productive response.  Shall Richard and I bundle up Matej's  
proposed updates to Richard's patch and contribute it back?  How do I  
go about contributing to Wicket?

Thanks,

--Ari

On Jun 30, 2008, at 3:05 AM, richardwilko wrote:

>
>
>
> Matej Knopp-2 wrote:
>>
>> What might be worth thinking of would be a terracotta based page  
>> store
>> (extending AbstractPageStore - class that implements the
>> (de)serialization magic). But even so it would still be necessary to
>> cluster http session.
>>
>
> Thats what kind of what I was thinking, and if we store this
> TerracottaPageStore in the httpsession, rather than on disk a la
> DiskPageStore, then we can just use terracotta to distribute the  
> httpsession
> as normal and everything *should* work
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18192381.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>

[Martijn Dashorst]

On Mon, Jun 30, 2008 at 5:11 PM, Ari Zilka <ari@...> wrote:
> a more productive response.  Shall Richard and I bundle up Matej's proposed
> updates to Richard's patch and contribute it back?  How do I go about
> contributing to Wicket?

Submit a patch to JIRA.

Martijn

[richardwilko]

Im looking into Implementing a new PageStore but I haven't quite got my head round everything yet.  My original change was much more simple, but was too simplistic for the general use case.

As for bundling everything up, it doesn't necessarily need integrating into wicket.  The classes could just be bundled into the wicket-terracotta integration module and then use the method visitor thing to force the use of the new session store (like the current one forces the use of httpsessionstore), however there is no real reason why it couldn't be integrated into main wicket code and would probably be more maintainable if it was.

Richard

Ari Zilka wrote:

Ok,

a more productive response.  Shall Richard and I bundle up Matej's  
proposed updates to Richard's patch and contribute it back?  How do I  
go about contributing to Wicket?

Thanks,

--Ari

On Jun 30, 2008, at 3:05 AM, richardwilko wrote:

>
>
>
> Matej Knopp-2 wrote:
>>
>> What might be worth thinking of would be a terracotta based page  
>> store
>> (extending AbstractPageStore - class that implements the
>> (de)serialization magic). But even so it would still be necessary to
>> cluster http session.
>>
>
> Thats what kind of what I was thinking, and if we store this
> TerracottaPageStore in the httpsession, rather than on disk a la
> DiskPageStore, then we can just use terracotta to distribute the  
> httpsession
> as normal and everything *should* work
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18192381.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>

[John Patterson]

That would be bloody brilliant!  I am planning on using TC and Wicket very soon but assumed the integration was already working.  This work will save me a lot of pain.

Ari Zilka wrote:

Ok,

a more productive response.  Shall Richard and I bundle up Matej's  
proposed updates to Richard's patch and contribute it back?  How do I  
go about contributing to Wicket?

Thanks,

--Ari

On Jun 30, 2008, at 3:05 AM, richardwilko wrote:

>
>
>
> Matej Knopp-2 wrote:
>>
>> What might be worth thinking of would be a terracotta based page  
>> store
>> (extending AbstractPageStore - class that implements the
>> (de)serialization magic). But even so it would still be necessary to
>> cluster http session.
>>
>
> Thats what kind of what I was thinking, and if we store this
> TerracottaPageStore in the httpsession, rather than on disk a la
> DiskPageStore, then we can just use terracotta to distribute the  
> httpsession
> as normal and everything *should* work
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18192381.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>

[richardwilko]

It does kind of work,  If you use that class on my blog and dont have references to pages inside all other pages then things will probably be fine, assuming that the httpsessionstore doesn't cause you any problems.  Our site is currently running at ~4500 active clustered sessions : )

Having said that I am making progress with the new session store and will probably have a tested version by the end of the week, depending on how busy I am.

Richard



That would be bloody brilliant!  I am planning on using TC and Wicket very soon but assumed the integration was already working.  This work will save me a lot of pain.

Ari Zilka wrote:

Ok,

a more productive response.  Shall Richard and I bundle up Matej's  
proposed updates to Richard's patch and contribute it back?  How do I  
go about contributing to Wicket?

Thanks,

--Ari

On Jun 30, 2008, at 3:05 AM, richardwilko wrote:

>
>
>
> Matej Knopp-2 wrote:
>>
>> What might be worth thinking of would be a terracotta based page  
>> store
>> (extending AbstractPageStore - class that implements the
>> (de)serialization magic). But even so it would still be necessary to
>> cluster http session.
>>
>
> Thats what kind of what I was thinking, and if we store this
> TerracottaPageStore in the httpsession, rather than on disk a la
> DiskPageStore, then we can just use terracotta to distribute the  
> httpsession
> as normal and everything *should* work
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18192381.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>

[John Patterson]

Much appreciated!

richardwilko wrote:

It does kind of work,  If you use that class on my blog and dont have references to pages inside all other pages then things will probably be fine, assuming that the httpsessionstore doesn't cause you any problems.  Our site is currently running at ~4500 active clustered sessions : )

Having said that I am making progress with the new session store and will probably have a tested version by the end of the week, depending on how busy I am.

Richard

[Matej Knopp-2]

Forgive me my ignorance :)

I'm just still not sure what benefits does clustering wicket
application with TC brings over regular session replication. And, is
it not possible to setup TC to acts just like regular session
replication (with serialization of modified session properties, etc.)

-Matej

On Mon, Jun 30, 2008 at 7:19 PM, John Patterson <jdp2000@...> wrote:

>
> Much appreciated!
>
>
> richardwilko wrote:
>>
>> It does kind of work,  If you use that class on my blog and dont have
>> references to pages inside all other pages then things will probably be
>> fine, assuming that the httpsessionstore doesn't cause you any problems.
>> Our site is currently running at ~4500 active clustered sessions : )
>>
>> Having said that I am making progress with the new session store and will
>> probably have a tested version by the end of the week, depending on how
>> busy I am.
>>
>> Richard
>>
>
> --
> View this message in context: http://www.nabble.com/Terracotta-integration-tp18168616p18200240.html
> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>
>

[Ari Zilka]

Well,

Terracotta session replication is far more scalable, and easy to  
manage the alternatives.

That said, Wicket has the disk-based solution which I do not  
understand yet.  So I can't say if Wicket is better w/ disk or with  
TC, but my guess is with TC because you can then share more than just  
Wicket state.  You can do Hibernate 2nd level cache, EHCache, and  
your own POJOs, all in the same impl.

So TC seems worth finishing up, no?

--Ari

On Jun 30, 2008, at 11:48 AM, Matej Knopp wrote:

> Forgive me my ignorance :)
>
> I'm just still not sure what benefits does clustering wicket
> application with TC brings over regular session replication. And, is
> it not possible to setup TC to acts just like regular session
> replication (with serialization of modified session properties, etc.)
>
> -Matej
>
> On Mon, Jun 30, 2008 at 7:19 PM, John Patterson <jdp2000@...>  
> wrote:
>>
>> Much appreciated!
>>
>>
>> richardwilko wrote:
>>>
>>> It does kind of work,  If you use that class on my blog and dont  
>>> have
>>> references to pages inside all other pages then things will  
>>> probably be
>>> fine, assuming that the httpsessionstore doesn't cause you any  
>>> problems.
>>> Our site is currently running at ~4500 active clustered sessions : )
>>>
>>> Having said that I am making progress with the new session store  
>>> and will
>>> probably have a tested version by the end of the week, depending  
>>> on how
>>> busy I am.
>>>
>>> Richard
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/Terracotta- 
>> integration-tp18168616p18200240.html
>> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>>
>>

[Eelco Hillenius]

> TC because you can then share more than just Wicket state.  You can do
> Hibernate 2nd level cache, EHCache, and your own POJOs, all in the same
> impl.

That is definitively a big pro for me. A one stop shop solution for
your clustering needs :-). And it comes with tools, source and
professional support options.

Eelco

[Matej Knopp-2]

Okay,

please don't confuse this with an attempt to flamewar, I'm just curious.

During "regular" session replication what basically happens is that
the modified session properties are serialized and sent across cluster
to backing node(s). Now when this message is being sent, Wicket has
already serialized the page to be stored in DiskPageStore and it uses
the serialization data, so there is very little extra serialization
done in clustered environment. After that the message is sent to the
target node and deserialized. Against, most of the message is not
really deserialized, it is just store in DiskPageStore (which is very
quick). The only deserialized object is the Session object, which is
usually couple of hundreds of bytes big.

I'm really curious in which aspect Terracotta makes the clustering
scale better.

This is roughly the purpose of DiskPageStore. Wicket only keeps the
last version of the last page instance in session. All previous
versions/instances are stored on Disk. SessionLevelCacheSessionStore +
DiskPageStore take care of this. There is a reason for not keeping the
pages in memory. It's mostly to conserve session usage. Also the vast
majority of pages kept in session store is never retrieved, as it only
happens when clicking back-button or using application in multiple
tabs/windows. So DiskPageStore is not a cache, even though lot of
people confuse it with one. It's a storage for previous pages, and
it's mostly write only.

That said, I also don't see the point of holding previous page
instances (even serialized) in HTTP session because the instances are
rarely used. But maybe TC overflow to disk can do better job than
diskpagestore and regular servlet containers, so I'm definitely not
against having proper terracotta integration.

-Matej



On Mon, Jun 30, 2008 at 8:54 PM, Ari Zilka <ari@...> wrote:

> Well,
>
> Terracotta session replication is far more scalable, and easy to manage the
> alternatives.
>
> That said, Wicket has the disk-based solution which I do not understand yet.
>  So I can't say if Wicket is better w/ disk or with TC, but my guess is with
> TC because you can then share more than just Wicket state.  You can do
> Hibernate 2nd level cache, EHCache, and your own POJOs, all in the same
> impl.
>
> So TC seems worth finishing up, no?
>
> --Ari
>
> On Jun 30, 2008, at 11:48 AM, Matej Knopp wrote:
>
>> Forgive me my ignorance :)
>>
>> I'm just still not sure what benefits does clustering wicket
>> application with TC brings over regular session replication. And, is
>> it not possible to setup TC to acts just like regular session
>> replication (with serialization of modified session properties, etc.)
>>
>> -Matej
>>
>> On Mon, Jun 30, 2008 at 7:19 PM, John Patterson <jdp2000@...> wrote:
>>>
>>> Much appreciated!
>>>
>>>
>>> richardwilko wrote:
>>>>
>>>> It does kind of work,  If you use that class on my blog and dont have
>>>> references to pages inside all other pages then things will probably be
>>>> fine, assuming that the httpsessionstore doesn't cause you any problems.
>>>> Our site is currently running at ~4500 active clustered sessions : )
>>>>
>>>> Having said that I am making progress with the new session store and
>>>> will
>>>> probably have a tested version by the end of the week, depending on how
>>>> busy I am.
>>>>
>>>> Richard
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Terracotta-integration-tp18168616p18200240.html
>>> Sent from the Wicket - Dev mailing list archive at Nabble.com.
>>>
>>>
>
>

[Ari Zilka]

No flames here. My point is the same as yours. While tc outruns container-builtin replication wicket is not using builtin. We do not do beteer than your impl.

I think would be good to have a one stop shop for clustering any and all objects when tc is present.

Richard is just going to mimic diskpagestore as you suggest.

Thanks. Very helpful.

--
Sent from my handheld

[Message delivered by NotifyLink]

----------Original Message----------

From: "Matej Knopp" <matej.knopp@...>
Sent: Mon, June 30, 2008 1:04 PM
To: dev@...
Subject: Re: Terracotta integration


Okay,

please don't confuse this with an attempt to flamewar, I'm just curious.

During "regular" session replication what basically happens is that
the modified session properties are serialized and sent across cluster
to backing node(s). Now when this message is being sent, Wicket has
already serialized the page to be stored in DiskPageStore and it uses
the serialization data, so there is very little extra serialization
done in clustered environment. After that the message is sent to the
target node and deserialized. Against, most of the message is not
really deserialized, it is just store in DiskPageStore (which is very
quick). The only deserialized object is the Session object, which is
usually couple of hundreds of bytes big.

I'm really curious in which aspect Terracotta makes the clustering
scale better.

This is roughly the purpose of DiskPageStore. Wicket only keeps the
last version of the last page instance in session. All previous
versions/instances are stored on Disk. SessionLevelCacheSessionStore +
DiskPageStore take care of this. There is a reason for not keeping the
pages in memory. It's mostly to conserve session usage. Also the vast
majority of pages kept in session store is never retrieved, as it only