Metakit mapped-file bug fix

[jcw]

Below is a bug fix which I have applied to the Metakit CVS head.

This may have been the cause of a few reports I have had over the  
years, which I could never reproduce reliably.  I do not know how the  
bug manifests itself exactly (Max reported new[] returning a pointer  
to inaccessible memory in c4_Column::CopyData),
but a problem this deep in the Metakit code base leads me to  
recommend updating to the CVS head where possible.  The only case  
when this bug does definitely not affect MK is if you use it with  
memory-mapping disabled.

I have also applied a bug fix to CVS which I recently discovered when  
re-opening a datafile and committing again several times (see the  
08-01 entry in CHANGES).

Lastly, I decided to revert to maintaining the CHANGES file manually  
again, instead of pointing to the possibly long and sometimes  
confusing timeline generated by CVStrac.

Will wrap this all up into a new Metakit 2.4.9.6 release if no-one  
else brings up other issues in the next week or so.

-jcw



Begin forwarded message:

> From: Max Kaehn
> Date: October 30, 2006 19:34:23 GMT+01:00
> To: jcw@...
> Subject: A very subtle Metakit bug, with fix
>
> Ran into an interesting one in column.cpp; in c4_Column::UsesMap(),
> there's a comparison:
>
>          ptr_ - Strategy()._mapStart < Strategy()._dataSize;
>
> This can go wrong in the following circumstance (which I have only
> observed intermittently on Linux 2.6 using gcc 3.3.3):
>
> 1.  mmap puts a file at a low address like 0x100000.
> 2.  malloc puts a buffer at a high address like 0xb8000000.
> 3.  The difference between a malloc'd address and the mmap'd
>     address is more than 2 gigabytes, so it comes out negative.
>     This is, of course, less than Strategy()._dataSize, so
>     c4_Column::UsesMap() returns true for a buffer that
>     doesn't actually use the map.
>
> The fix:
>          ptr_ < (Strategy()._mapStart + Strategy()._dataSize);
>
> Thanks for all your work on the database.

_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

[Ralf Schmitt-2]

Jean-Claude Wippler wrote:
> Below is a bug fix which I have applied to the Metakit CVS head.
>

Thanks.

>
> Will wrap this all up into a new Metakit 2.4.9.6 release if no-one else
> brings up other issues in the next week or so.
>

- metakit's python bindings won't compile on 64 bit platforms with
python 2.5 (see http://systemexit.de/repo/metakit?cs=9bed59f5a499 for a
fix).
- there's also a memleak in the python bindings as noted by zooko in
january (http://www.equi4.com/pipermail/metakit/2006-January/002170.html)
and by brian warner a few weeks ago:
http://www.equi4.com/pipermail/metakit/2006-October/002229.html

- Ralf

_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

[jcw]

On Oct 31, 2006, at 13:16, Ralf Schmitt wrote:

>> Will wrap this all up into a new Metakit 2.4.9.6 release if no-one  
>> else
>> brings up other issues in the next week or so.
>>
>
> - metakit's python bindings won't compile on 64 bit platforms with
> python 2.5 (see http://systemexit.de/repo/metakit?cs=9bed59f5a499 
> for a
> fix).

Integrated with cvs HEAD, thx.

> - there's also a memleak in the python bindings as noted by zooko in
> january (http://www.equi4.com/pipermail/metakit/2006-January/ 
> 002170.html)

See the follow-up: the unbounded mem use came from not doing any  
commits.

> and by brian warner a few weeks ago:
> http://www.equi4.com/pipermail/metakit/2006-October/002229.html

I've included these changes as well, thank you Brian.

Building via "sh configure --with-python=/usr" seems to be ok, but  
the now
more common "cd python; python setup.py build" method fails:

        $ python setup.py build
        Traceback (most recent call last):
          File "setup.py", line 166, in ?
            metakit = sys.modules['metakit']
        KeyError: 'metakit'

If someone wants to improve the Python build structure, please please  
do so - we've been balancing between autoconf and distutils for some  
time now, and unfortunately this seems to be a source of confusion  
and what's worse: failed builds.  See also:
        http://www.equi4.com/cgi-bin/cvstrac/metakit/tktview?tn=12

I don't understand why the python/setup.py file has to be so complex  
- but I'm not well versed in this and very reluctant to start  
changing it myself.

-jcw


_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

[Ralf Schmitt-2]

Jean-Claude Wippler wrote:
> On Oct 31, 2006, at 13:16, Ralf Schmitt wrote:
>>
>> - metakit's python bindings won't compile on 64 bit platforms with
>> python 2.5 (see http://systemexit.de/repo/metakit?cs=9bed59f5a499 for a
>> fix).
>
> Integrated with cvs HEAD, thx.

I've forgot another change (not related to python 2.5, but 64 bit
platforms):
http://systemexit.de/repo/metakit?cs=f8f1ef2f6099

>
>> - there's also a memleak in the python bindings as noted by zooko in
>> january (http://www.equi4.com/pipermail/metakit/2006-January/002170.html)
>
> See the follow-up: the unbounded mem use came from not doing any commits.
>
>> and by brian warner a few weeks ago:
>> http://www.equi4.com/pipermail/metakit/2006-October/002229.html
>
> I've included these changes as well, thank you Brian.

I thought they were the same, but I might be wrong.

>
> Building via "sh configure --with-python=/usr" seems to be ok, but the now
> more common "cd python; python setup.py build" method fails:
>
>     $ python setup.py build
>     Traceback (most recent call last):
>       File "setup.py", line 166, in ?
>         metakit = sys.modules['metakit']
>     KeyError: 'metakit'
>
> If someone wants to improve the Python build structure, please please do
> so - we've been balancing between autoconf and distutils for some time
> now, and unfortunately this seems to be a source of confusion and what's
> worse: failed builds.  See also:
>     http://www.equi4.com/cgi-bin/cvstrac/metakit/tktview?tn=12
>
> I don't understand why the python/setup.py file has to be so complex -
> but I'm not well versed in this and very reluctant to start changing it
> myself.
>

I have a scons file which builds the python bindings (and also replaces
the autoconf tests). This is much better for development (as it tracks
dependencies). But anyway, this isn't the standard python method of
building an extension.


- Ralf
_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

[Ralf Schmitt-2]

Ralf Schmitt wrote:
>
> I've forgot another change (not related to python 2.5, but 64 bit
> platforms):
> http://systemexit.de/repo/metakit?cs=f8f1ef2f6099
>

seems like you already fixed that one. sorry for the noise.



_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

[jcw]

Ralf Schmitt wrote:

> I've forgot another change (not related to python 2.5, but 64 bit
> platforms): http://systemexit.de/repo/metakit?cs=f8f1ef2f6099

Merged with cvs HEAD, thanks.  It's a better fix than the one I had.

-jcw


_____________________________________________
Metakit mailing list  -  Metakit@...
http://www.equi4.com/mailman/listinfo/metakit