Replication

Mike Boom wrote:
>
> I hope this is an appropriate place to ask this question. Is there a
> plan in the future for replication to provide readable replicas, or
> even better multiple masters?  If so what kind of time line are we
> looking at for completion?  Thanks.
>
Hi Mike,

I have not seen a lot of work on replication recently, other than
stabilization of the existing functionality and tests. Probably, the
original contributors would be happy to coach someone who wanted to
extend Derby replication.

Regards,
-Rick

I may want to add master/master replication or at least master/readonly
replication.  How would I find out who the original contributors would
be?

-----Original Message-----
From: Richard.Hillegas@... [mailto:Richard.Hillegas@...]
Sent: Monday, July 21, 2008 12:42 PM
To: derby-dev@...
Subject: Re: Replication

Mike Boom wrote:
>
> I hope this is an appropriate place to ask this question. Is there a
> plan in the future for replication to provide readable replicas, or
> even better multiple masters?  If so what kind of time line are we
> looking at for completion?  Thanks.
>
Hi Mike,

I have not seen a lot of work on replication recently, other than
stabilization of the existing functionality and tests. Probably, the
original contributors would be happy to coach someone who wanted to
extend Derby replication.

Regards,
-Rick

Hi Mike,

Currently the replication functionality is not a pluggable module but is rather strongly linked with the store
layer. For example LogToFile has been actually modified to catch the log records and store them in the log
buffer in addition to storing them on the master. If you are thinking about implementing a synchronous or
a synchronous (but read-only masters, I think in this case also you would be constrained to commit a transaction
only after all the masters sync), you would do great to think of how this can be implemented as a separate module
rather than coupling it tightly with the store as is happening now.

You could also think about convincing the community about the advantages a synchronous replication scheme might
bring to Derby over the existing asynchronous scheme and probably think about modifying existing implementation
to work in a sychronous way, or atleast think about how the existing functionality can be bundled as a separate unit.

Some good places to start would be

http://issues.apache.org/jira/browse/Derby-2872

http://wiki.apache.org/db-derby/ReplicationWriteup

You could also look at some synchronous replication products listed here

http://wiki.apache.org/db-derby/UsesOfDerby

looking at them would probably help you what type of an end product you would expect.

There was an effort from a student to provide hot standby functionality to derby, you can find the paper and related work here

https://issues.apache.org/jira/browse/DERBY-2852

I will help you in whatever way I can in this effort, but I may not actually be operation in synchronous mode ;) , I promise
asynchronous read-only, commit-maybe, support to your effort :) .

But your proposal looks interesting, please do remember to keep the community informed about your ideas, efforts and problems
constantly. I have noticed that the community appreciates continuous feedback on issues.

Narayanan
Mike Boom wrote:

I may want to add master/master replication or at least master/readonly
replication.  How would I find out who the original contributors would
be?

-----Original Message-----
From: Richard.Hillegas@... [Richard.Hillegas@...] 
Sent: Monday, July 21, 2008 12:42 PM
To: derby-dev@...
Subject: Re: Replication

Mike Boom wrote:
  

I hope this is an appropriate place to ask this question. Is there a 
plan in the future for replication to provide readable replicas, or 
even better multiple masters?  If so what kind of time line are we 
looking at for completion?  Thanks.

    

Hi Mike,

I have not seen a lot of work on replication recently, other than 
stabilization of the existing functionality and tests. Probably, the 
original contributors would be happy to coach someone who wanted to 
extend Derby replication.

Regards,
-Rick

  

Hi Mike,

Narayanan has already sent a pointer to the main replication JIRA:
http://issues.apache.org/jira/browse/Derby-2872 Besides Narayanan, the
other major contributors to that effort were Jørgen Løland and Øystein
Grøvlen.

Regards,
-Rick

Mike Boom wrote:

V Narayanan wrote: I would also suggest looking at adding a new replication module rather
than change the existing code.  The system can be set up such that the
new replication module can extend the existing functionality to achieve
the additional work necessary.  It was how I had expected a hot standby
to be implemented.

I think a read only replicant could be implemented using a lot of
existing functionality.  With
changes that did:
1) add replication module(s)
2) module to extend log buffer writes in some way such that log records
are sent to read only replicant some how.  How and how often depend on
how current you need the read only replicant.  You could probaby even
do this with no in line changes to log writing and just read the log
files from disk along with assuring the db is in backup mode which
always keeps all log files.  This mode would be interesting as it would
inpact the master performance as little as possible (except for possible
I/O throughput to log device).
3) add new module that applies log records and thus changes to the db,
but does not allow any external changes.  To support read committed or
higher than some extra work may need to be done in this area to get
expected locks while applying log records.  Currently the code that does
this assumes it is boot time and need not get locks.  Basically this
code wants to be a copy of the code that is done during boot recovery
but handle applying the log records "online" vs. "offline".  This is
probably the most challenging bit.