Nabble - Metakit

This mailing list is being moved.

2007-03-18T14:53:18Z

For various reasons, not in the least the fact that the amount of
email spam sent to the www.equi4.com server has reached quite a
bizarre level (the mail server *logs* are 5 Mb per day!), I have
decided to relocate this mailing list to another spot.

The new mailing list address is: metakit@...

The corresponding home page is: http://groups.google.com/group/
metakit

Instead of signing up all 230+ members of this list myself and making
mistakes w.r.t. duplicates, alternate addresses, and such, I'll defer
to everyone who wishes to continu tracking posts about metakit to
visit the above site and sign up yourself. It should be very quick
and easy to do.

The archives of metakit@... have been frozen and will remain
permanently at:
http://www.equi4.com/pipermail/metakit/

Email to this list will cease to work soon.

-jcw

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Error with python

2007-03-04T02:56:09Z

Hello, first I sorry my bad english.

I have this problem: when I execute this program on windows xp, python 2.4.3
break it and end

import metakit

print metakit.version

db = metakit.storage("ejemplo2.db",1)

#crear tablas y anadir filas

"""

vw = db.getas("dpto[departamento:I,descripcion:S]")

vw.append(departamento=1,descripcion="RRHH")

vw.append(departamento=2,descripcion="Comercial")

db.commit()

vw = db.getas("personal[id:I,nombre:S,departamento:I]")

vw.append(id=1,nombre="javier marquez", departamento=2)

vw.append(id=2,nombre="pepe lopez", departamento=2)

vw.append(id=3,nombre="ignacio garcia", departamento=1)

db.commit()

"""

vDpto = db.view('dpto')

vPersonal = db.view('personal')

for r in vPersonal: print r

out:

window with: python.exe ha detectado un problema y debe cerrarse (send error
etc)

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

_sopen (was: Corrupted database)

2007-02-15T08:53:10Z

Matt Oberle wrote:
> If we had spent all this time and energy working together to implement
> Pat's solution this would be a moot point.

by the way, does someone knows about _sopen() ?

-------- Original Message --------
Subject: Re: [Metakit] Corrupted database over network - WinNT
Date: Wed, 14 Feb 2007 16:09:13 +0100
From: Riccardo Cohen <rcohen@...>
Organization: articque
To: metakit@...

Hi Jean-Claude,
This is very clear, thank you. I really hope that vista corrected this windows bug. It is
not the only bug with windows networking, but this one is a real pain.
The problem now for us at least, is that we have many different way of using metakit.
sometimes we encapsulated in one of our class, wich make it easy to redefine a file
strategy, but sometimes we access directly for performance reason, and there are many
places where we open a storage.

Maybe I'm wrong, but there might be a very easy and effortless solution (for developers
that never have 2 C4_storage on the same file) :

in c4_FileStrategy::DataOpen(), I tried to replace :

fd = _wopen(wName, flags);

by :

fd=_wsopen(wName,flags,_SH_DENYRW);

Unfortunately it did not work, but really if someone know something about windows open()
function, this could help many people.

Thanks for any clue :)

Jean-Claude Wippler wrote:

> Thanks, Pat Thoyts for your detailed analysis and description of the
> workaround you found.
>
> Let me summarize as follows:
>
>> Metakit maps the file as
>> a read-only memory mapping and when it saves it re-serializes into the
>> underlying file. It then drops the current mapping and re-maps the
>> file. What I observed was that when it re-mapped it got the original
>> data and did not in fact re-map the newly written data at all.
>
> In other words, Windows does not guarantee a consistent view of the data
> it buffers for networked file systems: write something to a file, map it
> to memory *afterwards*, and apparently you can get stale data.
>
> If MK were to use r/w maps and push its data through there, then
> presumably it would work. That will not work in the case when the file
> has to be extended, though. Keep in mind that MK can write data to file
> for which partially comes from the r/o map. Furthermore, MK maps r/o
> deliberately, because it prevents stray pointer writes from damaging a
> MK datafile.
>
> So what we have today, is a design which works on Windows local disks,
> and on Unix/MacOSX both locally and on remote file systems (both NFS and
> SMB afaik).
>
> [From: <Hector@...>]
>> 1/ You confirm that metakit files have a serious design flaw that puts
>> at risk our data when accessed through a network.
>
> With all due respect, the "serious design flaw" is in Windows. Getting
> that fixed is beyond our control. We can only hope that XP or Vista get
> it right.
>
>> 2/ You give indications on the nature of the corruption, and how to
>> prevent it.
>
> Yes, and for that I am very grateful. In addition to Pat's most recent
> post, more information can be found at
> http://www.equi4.com/pipermail/metakit/2005-August/002112.html
>
> So the solution is to derive a c4_Strategy class which opens the file in
> exclusive mode. Pat has most generously handed everyone the solution on
> a silver platter. If you don't want to subclass c4_Strategy, then open
> the file yourself and pass it to MK. If you never need concurrent
> readers, use CreateFile and always open with dwAccessMode set to 0, then
> turn it into a FILE* with freopen and pass it to MK.
>
> I for one never use MK in r/w mode on remote file systems. Not just due
> to the above risks, but because locking also leads to all sorts of
> "Heisenberg" trouble. In my book, shared data use should use
> client/server solutions. That need only take a few dozen lines of code,
> depending on your tool choices.
>
> -jcw
>
>
> _____________________________________________
> Metakit mailing list - Metakit@...
> http://www.equi4.com/mailman/listinfo/metakit
>

--
Très cordialement,

Riccardo Cohen
ligne directe : 02-47-49-63-24
-------------------------------------------
Articque
http://www.articque.com
Moulin des Roches
37230 Fondettes - France
tel : 02-47-49-90-49
fax : 02-47-49-91-49
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

--
Très cordialement,

Riccardo Cohen
ligne directe : 02-47-49-63-24
-------------------------------------------
Articque
http://www.articque.com
Moulin des Roches
37230 Fondettes - France
tel : 02-47-49-90-49
fax : 02-47-49-91-49
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-15T08:32:36Z

If we had spent all this time and energy working together to implement
Pat's solution this would be a moot point.

Matt

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-15T05:46:04Z

Corrupted database over network - WinNT

2007-02-15T05:21:01Z

> You made the choice to use MK and you are responsible from your platform
> decisions.

When I chose Metakit, several year ago, it was an actively maintained
project. It's barely the case anymore. At that time, a corruption issue
wouldn't have been tolerated.
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-15T05:10:39Z

> I wouldn't write a single line of code for a person like you, even if you
> paid $100.000. You have been acting not educated (on open source), not
> tolerant and not professional (~polite).
>
> I think you should be responsible and apologize for your earlier posts
> before asking anything. You made the choice to use MK and you are
> responsible from your platform decisions.

Seriously, I can't expecting someone to fix this bug for 100$. If this bug
is fixed by someone someday, I'll thank him with 100$, that's different. Let
say I put it in the pot. I already regret to have made that offer. Sounds
like I'm asking something...

Reading this list, I could trace people losing their data up to 2002, but
the origin of the corruption wasn't identified. I thought it was my
contribution to locate the source of the bug (and I lost many hair in the
battle). Now that the corruption source is well identified, why not warn
people ? Why ?

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-15T03:31:19Z

>>>Hector@... said:
>
> I chose Metakit because:
> 1/ it is an excellent library that perfectly suits my needs.
> 2/ it's open source and the author has very good reputation of seriousness
> (hence my disappointment).
>

What you have run up against is the real evil of the proprietary
nature of the dominant desktop operating system. The bug, if that's
what it is, is in Windows: it does not cater for the generality of use
to which it may be put, which involves a lot of us in chasing round
finding fixes for our applications (and making damn sure we never
never never *trust* Windows to behave as expected). I just spent the
best part of a week fixing some of our own code that broke, after
several years of flawless use, because the customer started running
software built for Win2000 on XP SP2. Nothing to do with MetaKit,
same code on various Unices and everything from MS-DOS 2.0 to Win2000
has run with no such error for 18 years.

If I had had the source to Windows, that is what I would have patched.
As it is, Micro$oft attempt to make the world revolve around them, to
conform to their choices and pettifogging 'philosophy'. So I had to
write a workaround, which complicates my code unnecessarily.

My experience in general with Windows networking (and other aspects of
it as an OS) is that you should never trust it not to cache what you
would rather it did not cache, to cache what you expect it to cache,
or not to have undisclosed limitation, flaws and "features".

I tend to agree with jcw that data sharing over a network is best done
using a client-server approach: there are too many lines of code you
will never get to see in between otherwise, and trusting them without
checking is perilous.

And please cut the histrionics. This is a courteous mutual assistance
forum. Those of us who consume more than we are able to supply owe
at least a certain politesse, not to say gratitude, for the generosity
of others.

--
Steve Blinkhorn <steve@...>
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-15T03:12:58Z

> But in the real world, you have what you pay for.

I thought about it again, and I'll give 100$ to the person who'll fix this
bug.

I know this sum might seem ridiculous, but seriously it's the max I can
offer.

And if this bug is fixed, I'd like the patch to be integrated in Metakit
main source.

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-15T01:14:50Z

> Open source is mainly a community, but it is also risk (re-read
> the metakit license, in particular the line WITHOUT
> WARRANTY OF ANY KIND).

I have no problem with that. I never said that I would sue Metakit for
having lost my data.

> You could have used ctree+ or mysql embedded

I chose Metakit because:
1/ it is an excellent library that perfectly suits my needs.
2/ it's open source and the author has very good reputation of seriousness
(hence my disappointment).

> Even if I agree that it is a windows bug, I also agree that he should not
> have answered that.

Thank you.

> But in the real world, you have what you pay for.

I can't pay no dollar, so I don't ask anything to anyone. Except the warning
on the site, of course. It's not because you write "WITHOUT WARRANTY OF ANY
KIND" that you can play with people's data integrity.

> Now you are free to criticize J.C. but be aware that
> this choice is probably not very fair, and also probably
> not what will give you the solution.

As I already said, I know perfectly... but I indulged myself.

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-14T16:26:33Z

Suppose you choose yourself a car because it's advertized as powerful and
secure. You're driving your new car when suddenly the wheel is jammed and
you crash into a wall. The car manufacturer is informed of that issue, and
quietly answers that you must fix the problem yourself, it's not his fault
if the roads have turns and bends...

1) the car manufacturer is blatantly insulting you.
2) other uninformed persons will also crash themselves.
3) it's illogical that each driver should fix the problem for themselves,
because most drivers know nothing about mechanics, and before all because
the problem should be fixed at the car factory.

As I've said, I have no way to correct this Metakit bug. But I do care about
my user's data. I was thus forced to place big warnings on my site, and live
with it. All I ask is that Jean-Claude places big warnings on his site too,
and live with it.

Now I've said all I had to say. I won't reply anymore. Or this discussion
will never calm down.

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-14T14:20:18Z

> I'm very upset by your attitude. Jean-Claude Wippler, Pat Thoyts and many
> others
> put a great deal of their own time and effort into providing software
> products
> that the likes of you and I benefit from. They are also prompt, helpful
> and
> courteous in responding to design and usage issues.
>
> Be grateful for what you are given. If you don't like it, ask politely for
> assistance to modify it or decline it. Don't bite the hand that feeds you.

If you re-read my posts chronologically, you'll realize that I took the
angry tone only after Jean-Claude said it was a Windows bug, and he hoped
Microsoft would fix it. What a ridiculous statement is that ? If your
application has problems with Vista, you don't say it is Vista-ready. If
your program has problems with Windows, either you say so, either you fix
your program.

Before that, all I had asked was whether Jean-Claude thought he would fix
this bug in the future. No more.

Data corruption isn't something that can be taken lightly, is it ?

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-14T13:37:11Z

Quoting Hector@...:

> >From Metakit website:
> "This library is in active use in various commercial projects and products
> on millions of desktops."
>
> Why don't you add that people happen to LOOSE their precious data ?
>
> Angry ? Yes I am !
> _____________________________________________

Hector,

I'm very upset by your attitude. Jean-Claude Wippler, Pat Thoyts and many others
put a great deal of their own time and effort into providing software products
that the likes of you and I benefit from. They are also prompt, helpful and
courteous in responding to design and usage issues.

Be grateful for what you are given. If you don't like it, ask politely for
assistance to modify it or decline it. Don't bite the hand that feeds you.

Phil Bennett

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-14T09:02:42Z

>From Metakit website:
"This library is in active use in various commercial projects and products
on millions of desktops."

Why don't you add that people happen to LOOSE their precious data ?

Angry ? Yes I am !
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-14T09:01:38Z

>From Metakit website:
"This library is in active use in various commercial projects and products
on millions of desktops."

Why don't you add that people happen to use their precious data ?

Angry ? Yes I am !

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-14T08:44:55Z

> In my book, shared data use
> should use client/server solutions.

It's the first time I hear you've written a book... That's interesting.

It's not shared data, only 1 program opens the file. It just happens that
the file that is located on a NAS (Network Attached Storage)... There can't
be a server running on the NAS, or I'm missing something...

I ask for only 1 thing. Now that this issue has been clearly identified, you
MUST place a big warning on your site ! Because loosing one's data is not
acceptable. The 2 horrid weeks I've spent on that issue, I give you for free
!

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-14T08:27:40Z

> With all due respect, the "serious design flaw" is
> in Windows. Getting that fixed is beyond our control.
> We can only hope that XP or Vista get it right.

With all due respect too, and I didn't mean to offend you, if a Metakit file
gets corrupted on XP, which is by far the most used OS, I tend to think that
we can call it a bug... Why play on words ? Until now, that network
corruption was just a myth (from what I've read on this list), and nowhere
it is specified that Metakit doesn't work on XP networks, and that people
can lose their data. This issue has driven me crazy for nearly 2 weeks
before I could locate when and how the corruption happened. Unfortunately, I
wasn't smart enough to fix the bug. Of course, if I had fixed the bug, I
would have contributed the patch, because this kind of patch obviously must
be included inside Metakit's main code.

About Pat's boss: how honest is this to use an open source library, to
discover a bug, and to fix it for himself without telling everyone ?

About engaging Jean-Claude( or another developer) to produce the workaround:
if you know my finances, you'd know it is science-fiction.

All I could do is place a big warning on my site:
DON'T USE THROUGH A NETWORK !!!
DON'T USE THROUGH A NETWORK !!!
DON'T USE THROUGH A NETWORK !!!
DON'T USE THROUGH A NETWORK !!!

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-14T07:09:13Z

Hi Jean-Claude,
This is very clear, thank you. I really hope that vista corrected this windows bug. It is
not the only bug with windows networking, but this one is a real pain.
The problem now for us at least, is that we have many different way of using metakit.
sometimes we encapsulated in one of our class, wich make it easy to redefine a file
strategy, but sometimes we access directly for performance reason, and there are many
places where we open a storage.

Maybe I'm wrong, but there might be a very easy and effortless solution (for developers
that never have 2 C4_storage on the same file) :

in c4_FileStrategy::DataOpen(), I tried to replace :

fd = _wopen(wName, flags);

by :

fd=_wsopen(wName,flags,_SH_DENYRW);

Unfortunately it did not work, but really if someone know something about windows open()
function, this could help many people.

Thanks for any clue :)

Jean-Claude Wippler wrote:

Re: Corrupted database over network - WinNT

2007-02-14T05:50:00Z

Thanks, Pat Thoyts for your detailed analysis and description of the
workaround you found.

Let me summarize as follows:

> Metakit maps the file as
> a read-only memory mapping and when it saves it re-serializes into the
> underlying file. It then drops the current mapping and re-maps the
> file. What I observed was that when it re-mapped it got the original
> data and did not in fact re-map the newly written data at all.

In other words, Windows does not guarantee a consistent view of the
data it buffers for networked file systems: write something to a
file, map it to memory *afterwards*, and apparently you can get stale
data.

If MK were to use r/w maps and push its data through there, then
presumably it would work. That will not work in the case when the
file has to be extended, though. Keep in mind that MK can write data
to file for which partially comes from the r/o map. Furthermore, MK
maps r/o deliberately, because it prevents stray pointer writes from
damaging a MK datafile.

So what we have today, is a design which works on Windows local
disks, and on Unix/MacOSX both locally and on remote file systems
(both NFS and SMB afaik).

[From: <Hector@...>]
> 1/ You confirm that metakit files have a serious design flaw that
> puts at risk our data when accessed through a network.

With all due respect, the "serious design flaw" is in Windows.
Getting that fixed is beyond our control. We can only hope that XP
or Vista get it right.

> 2/ You give indications on the nature of the corruption, and how to
> prevent it.

Yes, and for that I am very grateful. In addition to Pat's most
recent post, more information can be found at http://www.equi4.com/
pipermail/metakit/2005-August/002112.html

So the solution is to derive a c4_Strategy class which opens the file
in exclusive mode. Pat has most generously handed everyone the
solution on a silver platter. If you don't want to subclass
c4_Strategy, then open the file yourself and pass it to MK. If you
never need concurrent readers, use CreateFile and always open with
dwAccessMode set to 0, then turn it into a FILE* with freopen and
pass it to MK.

I for one never use MK in r/w mode on remote file systems. Not just
due to the above risks, but because locking also leads to all sorts
of "Heisenberg" trouble. In my book, shared data use should use
client/server solutions. That need only take a few dozen lines of
code, depending on your tool choices.

-jcw

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-14T04:45:47Z

On 14/02/2007, at 9:22 PM, <Hector@...>
<Hector@...> wrote:

>> Hector, Matt,
>>
>> I won't be working on the problem for a while. So, if either of you
>> implement a suitable c4_strategy perhaps you would be kind enough
>> to share
>> your work and spare me some effort. I'll do likewise if I get
>> ahead of you.
>
> Well, though I'm a Metakit user, I'm far from being familiar with
> its internals. I had a look at it, and it looks beyond my skills.
> I wait for JCW to make a statement on the subject. I would be very
> surprised/disappointed if he was aware of a serious corruption bug
> and did nothing about it...

There is no serious corruption bug in Metakit - rather there is a bug
(or limitation) in Windows networking that is triggered by Metakit.

Now, the distinction might appear to be tenuous, but I put it to you
that you have at least three alternatives
- diagnose and produce a workaround (as Pat did)
- wait for someone to contribute a workaround
- engage Jean-Claude( or another developer) to produce the workaround

The bottom line is that you're using an open source product on a
proprietary platform and (unlike many many many people using Metakit
on other platforms) you're encountering problems due to the broken
nature of that platform. Whilst that is unfortunate, to tar Metakit
with that brush or imply that Jean-Claude is in any way obliged to
provide you with a workaround is (IMNSHO) neither appropriate nor
helpful.

Now that I've got that off my chest, let me say that I'm sure people
will help if they can (evidence Pat's detailed mailing on the
topic). I share your frustration in that there is much about Windows
networking that drives me crazy. But your alternatives are clear -
you either need patience or be willing to invest time and/or money to
provide a workaround in the timescale you need. That's life, I'm
afraid.

Regards

Steve

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-14T04:22:01Z

> Hector, Matt,
>
> I won't be working on the problem for a while. So, if either of you
> implement a suitable c4_strategy perhaps you would be kind enough to share
> your work and spare me some effort. I'll do likewise if I get ahead of
> you.

Well, though I'm a Metakit user, I'm far from being familiar with its
internals. I had a look at it, and it looks beyond my skills.
I wait for JCW to make a statement on the subject. I would be very
surprised/disappointed if he was aware of a serious corruption bug and did
nothing about it...

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-13T14:57:26Z

----- Original Message -----
From: <Hector@...>
To: "Pat Thoyts" <patthoyts@...>
Sent: Tuesday, February 13, 2007 11:56 PM
Subject: Re: [Metakit] Corrupted database over network - WinNT

> >... discovered that when accessed over network shares
>> we sometimes obtained corrupt matakit databases.
>
> Wow, what a precious testimony !
>
> 1/ You confirm that metakit files have a serious design flaw that puts at
> risk our data when accessed through a network.
> 2/ You give indications on the nature of the corruption, and how to
> prevent it.
>
>
> To JCW:
> Man, I understand you're very busy, but you've always taken corruption
> bugs seriously. With all the details that Pat gave, is there a chance that
> you'll fix this issue in the future, for the benefit of all of us who are
> not familiar enough with Metakit's guts ?
>
> Thanks.

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-13T13:29:42Z

<Hector@...> writes:

>Thank you very much for your help, Pat.
>
>Does it mean you made the simple test I described and nothing happened ?
>
>What did you change to c4_FileStrategy ?

I asked and it turns out I am not being given permission to post the
code for my c4_Win32FileStrategy implementation. Quite frankly I'm
rather disgusted with my bosses about this decision. If we are to
build upon an open source foundation the very least we could do is
share information freely. At least I can describe what was done
without explicit reference to the code. Maybe later I can clue them up
a bit.

We implemented a file format based upon metakit for storing
application specific data and discovered that when accessed over
network shares we sometimes obtained corrupt matakit databases. The
symptoms were generally swapped columns (the size data would appear in
the timestamp column and vice versa for instance). After a lot of
searching I was able to show the problem is to do with the handling of
memory mapped files over the network share.

I then examined carefully the way the data looked in the
ResetFileMapping function when saving. Metakit doesn't use memory
mapping the way that most applications do and I think this confuses
the windows network layer that handles this. Metakit maps the file as
a read-only memory mapping and when it saves it re-serializes into the
underlying file. It then drops the current mapping and re-maps the
file. What I observed was that when it re-mapped it got the original
data and did not in fact re-map the newly written data at all. I found
it useful to dump the mapped memory image to a sequence of temporary
files to compare the various products when testing this.

Now metakit by default doesn't provide much control over how a file
gets accessed. You pass in a filename and a mode (readonly or
readwrite) and it gets opened for you. If you look in src/fileio.c you
can find the DataOpen function and see it is using the C library to
open the file and uses fseek/fprintf/fread to manipulate the file. I
wrote a new implementation that eliminates the use of the C library
functions and instead allows us to use CreateFile,ReadFile, WriteFile
and SetFilePosition instead. The primary advantage of this is that
instead of having metakit open a filename we use CreateFile to open
the file and then have metakit attach to the file handle. In use it
goes something like:

HFILE hFile = CreateFile(....);
c4_Strategy *pStrategy = new c4_Win32FileStrategy();
if (!pStrategy->DataOpen(hFile, true)) {
return STG_E_ACCESSDENIED;
}
c4_Storage *pStorage = new c4_Storage(*m_pStrategy, false, 1);

This permits us to specify much more completely the Win32 flags and
permissions used to open the file. It also lets us provide our own
file prefix before the metakit data section which is another reason we
use this method.

Using the above we can open the file with the dwAccessMode set to 0
for network files or FILE_SHARE_READ for local files. We can also add
some flags -
FILE_ATTRIBUTE_NORMAL|FILE_FLAG_RANDOM_ACCESS |FILE_FLAG_WRITE_THROUGH
seems about right. We also use FILE_FLAG_OVERLAPPED but I don't think
that has anything to do with the corruption issues.

The key point for me was taking absolute control of how the file was
opened. If you have exclusive access to the file then you don't need
to worry about anyone else - virus checker or not - until you close
the file handle. Some of the other flags (overlapped or random access)
may or may not contribute to solving the problem, but I think I tested
a number of combinations before settling with exclusive access.

--
Pat Thoyts http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-13T12:47:03Z

Pat,
I would also be grateful if you could send me a copy of your
c4_FileStrategy.
Regards,
Matt Oberle

Pat,

I'm running a far less ambitious but similarly configured ( Windows network
) shared database and, despite arbitrating access, have experienced some of
the trouble you and Hector are talking about. I would be very grateful if
you would send me a copy of the c4_FileStrategy scheme that you have
implemented - it certainly sounds like you've cracked the problem.

Regards,
Phil Bennett

-----Original Message-----
From: metakit-bounces@... [mailto:metakit-bounces@...] On
Behalf
Of Pat Thoyts
Sent: 13 February 2007 15:28
To: Hector@...
Cc: metakit@...
Subject: Re: [Metakit] Corrupted database over network - WinNT

<Hector@...> writes:

>> As I mentioned in an earlier post -- the solution is to ensure that
>> metakit files on network shares are opened for exclusive access. You
>> can write your own c4_FileStrategy implementation to handle this and
>> use it in the c4_Storage constructor. The PathIsNetworkPath Win32 API
>> is also a useful function here.
>
>No, you can't prevent an antivirus from scanning your file. Just
>looking at the file attributes is enough to corrupt it (if it's
>followed by another commit). At least, it is what I observe with my
>program. If you too have an application that can commit to a metakit
>file through a network, please make the test and tell me if you
>observe the same corruption. Maybe it's my program's fault, but
>sincerely I doubt it.

I can't post you the whole lot but I can give you the re-implementation
of c4_FileStrategy that we use. We have some 32GB of data in over
18000 files on a Windows network shared storage and none of them are
corrupt and people do use them over the network share.

--
Pat Thoyts http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

--
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.441 / Virus Database: 268.17.37/682 - Release Date: 12/02/2007
13:23

--
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.441 / Virus Database: 268.17.37/682 - Release Date: 12/02/2007
13:23

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-13T08:03:59Z

Thank you very much for your help, Pat.

Does it mean you made the simple test I described and nothing happened ?

What did you change to c4_FileStrategy ?
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

RE: Corrupted database over network - WinNT

2007-02-13T07:50:16Z

Pat,

I'm running a far less ambitious but similarly configured ( Windows network
) shared database and, despite arbitrating access, have experienced some of
the trouble you and Hector are talking about. I would be very grateful if
you would send me a copy of the c4_FileStrategy scheme that you have
implemented - it certainly sounds like you've cracked the problem.

Regards,
Phil Bennett

-----Original Message-----
From: metakit-bounces@... [mailto:metakit-bounces@...] On Behalf
Of Pat Thoyts
Sent: 13 February 2007 15:28
To: Hector@...
Cc: metakit@...
Subject: Re: [Metakit] Corrupted database over network - WinNT

<Hector@...> writes:

Re: Corrupted database over network - WinNT

2007-02-13T07:28:16Z

<Hector@...> writes:

Re: Corrupted database over network - WinNT

2007-02-12T15:40:58Z

> As I mentioned in an earlier post -- the solution is to ensure that
> metakit files on network shares are opened for exclusive access. You
> can write your own c4_FileStrategy implementation to handle this and
> use it in the c4_Storage constructor. The PathIsNetworkPath Win32 API
> is also a useful function here.

No, you can't prevent an antivirus from scanning your file. Just looking at
the file attributes is enough to corrupt it (if it's followed by another
commit). At least, it is what I observe with my program. If you too have an
application that can commit to a metakit file through a network, please make
the test and tell me if you observe the same corruption. Maybe it's my
program's fault, but sincerely I doubt it.

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: Corrupted database over network - WinNT

2007-02-12T15:09:06Z

<Hector@...> writes:

>Hi,
>
>I think I have located the origin of this long-lasting corruption problem.
>Here is what I have observed with my application on an XP network:
>
>1/ the program commits to a metakit file located on the network. It's OK.
>2/ a process slightly touches the file. It can be as slight as an anti-virus
>who scans the file, or a person who checks the properties of the file
>(e.g last modification time), etc...
>3/ if the program commits once again, then the file is corrupted !!!
>
>This only happens whith files on a network. I can't explain any more
>than this, but it gives me the shivers to think how fragile my metakit
>files are :(
>

As I mentioned in an earlier post -- the solution is to ensure that
metakit files on network shares are opened for exclusive access. You
can write your own c4_FileStrategy implementation to handle this and
use it in the c4_Storage constructor. The PathIsNetworkPath Win32 API
is also a useful function here.

--
Pat Thoyts http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Corrupted database over network - WinNT

2007-02-11T13:58:17Z

Hi,

I think I have located the origin of this long-lasting corruption problem.
Here is what I have observed with my application on an XP network:

1/ the program commits to a metakit file located on the network. It's OK.
2/ a process slightly touches the file. It can be as slight as an anti-virus
who scans the file, or a person who checks the properties of the file (e.g
last modification time), etc...
3/ if the program commits once again, then the file is corrupted !!!

This only happens whith files on a network. I can't explain any more than
this, but it gives me the shivers to think how fragile my metakit files are
:(

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: multiple readers and multiple writers

2007-02-10T19:35:06Z

metakit supports one writer and multiple readers with the caveat if
the writers writes to the same table a reader is "reading" the reader
becomes invalid. I.e. the data is not what you would expect.

Brian

On 2/10/07, Eric S. Johansson <esj@...> wrote:

> I forget if I've asked this before and I apologize if I did. Does
> metakit use a shared memory model of operation and support multiple
> readers and multiple writers? My application is drowning in little dbm
> files and I'd like to unify things and take advantage of keeping data in
> memory whenever possible. as I told a friend of mine:
>
> each user has a fast white list, a cache for green, yellow, red traps.
>
> for site wide, I have a verify cache as well as reputation and rate of
> arrival. rate of arrival and reputation will need to be accessible from
> multiple systems.
>
> when processing messages and done, I open fast white list (user
> specific), verify cache, reputation, and and one of the green, yellow,
> red traps. Right now with DBMS, most of the transactions are
> open/lock/rw/close.
>
> I know I keep circling around the drain on this topic but I think that's
> because back of my brain is telling me that my current solution is
> wrong. I'm just hoping when my friend pops over to talk about databases
> that I can stop circling. :-)
> ---eric
>
>
> --
> Speech-recognition in use. It makes mistakes, I correct some.
> _____________________________________________
> Metakit mailing list - Metakit@...
> http://www.equi4.com/mailman/listinfo/metakit
>

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

multiple readers and multiple writers

2007-02-10T11:14:20Z

I forget if I've asked this before and I apologize if I did. Does
metakit use a shared memory model of operation and support multiple
readers and multiple writers? My application is drowning in little dbm
files and I'd like to unify things and take advantage of keeping data in
memory whenever possible. as I told a friend of mine:

each user has a fast white list, a cache for green, yellow, red traps.

for site wide, I have a verify cache as well as reputation and rate of
arrival. rate of arrival and reputation will need to be accessible from
multiple systems.

when processing messages and done, I open fast white list (user
specific), verify cache, reputation, and and one of the green, yellow,
red traps. Right now with DBMS, most of the transactions are
open/lock/rw/close.

I know I keep circling around the drain on this topic but I think that's
because back of my brain is telling me that my current solution is
wrong. I'm just hoping when my friend pops over to talk about databases
that I can stop circling. :-)
---eric

--
Speech-recognition in use. It makes mistakes, I correct some.
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

NEWBIE: How do I retrieve the row index?

2007-02-10T00:24:11Z

Consider this code:

#Create a database:

> import metakit
> db = metakit.storage("passwordfile.mk",1)
> #Create a view (this is the Metakit term for "table"):
> vw = db.getas("users[username:S,password:S]")
> #get user name to add
> user = raw_input('User Name to Add: ')
> # check and see if it exists in datafile already
> c_vw = vw.select(username=user)
> if c_vw:
> for r in c_vw:
> ix = vw.find(username=user, start=2) #=================>>>>wanted
> to get row index here, is broken
> print r.username + ' already exists in database at: ', ix
> else:
> # get a password for the new user
> passw = raw_input('Password for that user: ')
> #Add a row (this is the Metakit term for "record"):
> idx = vw.append(username=user,password=passw)
> print 'Base0 index position where record was inserted: ' + str(idx)
> #Commit the changes to file:
> db.commit()
> #Show a list of all users
> for r in vw: print r.username, r.password

#==============>>>>>>>>>>>>>>>>>>>>>>END SCRIPT

I have been through all the documentation that I could find, and was unable
> to find out how to get the row index.

Could someone give a push in the right direction??

TIA :-)
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

Re: error while database is on windows file server

2007-02-09T04:01:02Z

"Florent Lejeune" <flejeune@...> writes:

>Hi
>
>We use Metakit to store miscellaneous information for our programe. It works on windows,
>macosx and linux.
>It happens that when the database is created on the windows network, we have some problem
>with the windows application (we use metakit-2.4.9.6).
>
>Below you will find a program that creates a file, adds rows, and retrieves information.
>
>The Find function doesn't return anything and a manuel search also gives no result. In our
>main program, the Find() function remains in an infinite loop.
>It works fine with other plateform combinations : from windows client to simple windows
>sharing, from macosx client to windows file server and from macosx client to macosx apple share
>file server.
>
>Thanks for any suggestion about this.

You don't say what kind of problem you have with the network files but
I will make a guess that it sometimes produces corrupt metakit
databases. Metakit works by mapping the file into memory as a
read-only memory mapped file and when it saves any changes it
serializes the in-memory version to the file, unmaps and re-maps the
database. We found that when the file is on a network share the re-map
step doesn't re-map the new file, but instead maps the old image and
breaks everything.

The simplest solution I found was to ensure that if the file is mapped
to a network share that we open the windows file for exclusive
access. There are likely some longer posts about my bug hunt in the
archive for this list if this is relevant.

--
Pat Thoyts http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD

_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit

error while database is on windows file server

2007-02-09T01:51:58Z

Hi

We use Metakit to store miscellaneous information for our programe. It works on windows,
macosx and linux.
It happens that when the database is created on the windows network, we have some problem
with the windows application (we use metakit-2.4.9.6).

Below you will find a program that creates a file, adds rows, and retrieves information.

The Find function doesn't return anything and a manuel search also gives no result. In our
main program, the Find() function remains in an infinite loop.
It works fine with other plateform combinations : from windows client to simple windows
sharing, from macosx client to windows file server and from macosx client to macosx apple share
file server.

Thanks for any suggestion about this.

PS: our windows file server is a macosx with Samba. we compile with visual studio 2005

#include <stdio.h>
#include <stdlib.h>
#include <conio.h>
#include <string.h>
#include <mk4.h>

void main(int argc, char **argv)
{
char dbpath[100]="";
printf("Choose a db destination :\n\n");
printf("1 - Local\n");
printf("2 - Network : Windows file server\n");
printf("3 - Network : Windows sharing\n\n");

switch (getche())
{
case '1':
strcpy(dbpath,"c:\\zdrop\\metakit_tutorial.db");
break;
case '2':
strcpy(dbpath,"\\\\Noe\\Developpement\\metakit_tutorial.db");
break;
case '3':
strcpy(dbpath,"\\\\Ordiludovic\\partageArticque\\metakit_tutorial.db");
break;
default:
break;
}
printf("\n");
if (strlen(dbpath)>0)
{
fflush(stdin);
FILE *fp=NULL;
// remove if present
if ((fp=fopen(dbpath,"r"))!=NULL)
{
fclose(fp);
unlink(dbpath);
}
// create db
c4_Storage database(dbpath, true);
// add datas
c4_View maps=database.GetAs("maps[mid:S,mwidth:I,mheight:I,mpath:S]");
c4_StringProp mid("mid"),mpath("mpath");
c4_IntProp mwidth("mwidth"),mheight("mheight");
c4_Row maprow;
mid(maprow)="A";
mwidth(maprow)=10;
mheight(maprow)=10;
mpath(maprow)="/home/MapA.vxf";
maps.Add(maprow);
mid(maprow)="B";
mwidth(maprow)=20;
mheight(maprow)=20;
mpath(maprow)="/home/MapB.vxf";
maps.Add(maprow);
mid(maprow)="C";
mwidth(maprow)=30;
mheight(maprow)=30;
mpath(maprow)="/home/MapC.vxf";
maps.Add(maprow);
// save
database.Commit();

// add more datas
c4_View datas=database.GetAs("datas[did:S,dmapid:S,dpath:S]");
// add data
c4_StringProp did("did"),dmapid("dmapid"),dpath("dpath");
c4_Row datarow;
did(datarow)="1";
dmapid(datarow)="A";
dpath(datarow)="/home/A1.txt";
datas.Add(datarow);
did(datarow)="2";
dmapid(datarow)="A";
dpath(datarow)="/home/A2.txt";
datas.Add(datarow);
did(datarow)="3";
dmapid(datarow)="B";
dpath(datarow)="/home/B1.txt";
datas.Add(datarow);
did(datarow)="4";
dmapid(datarow)="B";
dpath(datarow)="/home/B2.txt";
datas.Add(datarow);
did(datarow)="5";
dmapid(datarow)="C";
dpath(datarow)="/home/C1.txt";
datas.Add(datarow);
// save
database.Commit();
// Find data
long idxsearch=-1;
c4_Row findrow;
dmapid(findrow)="B";
while(1)
{
idxsearch=datas.Find(findrow,idxsearch+1);
if (idxsearch>=0)
printf("DATA %s has dmapid=%s\n",
(const char*)dpath(datas[idxsearch]),
(const char*)dmapid(datas[idxsearch]));
else
break;
}
// manuel search
c4_Row strrow;
const char*strdata;
const char*expr="ome/B";
for (int idx=0;idx<datas.GetSize();idx++)
{
strdata=(const char*)dpath(datas[idx]);
if (strstr(strdata,expr)!=NULL)
printf("map %s has the string '%s' in the path\n",strdata,expr);
}

}
printf("Test over\n");
getch();
}

--
Florent Lejeune
Développeur

Articque
Moulin des Roches
37230 Fondettes
France
web = http://www.articque.com
tel: +33 02 47 49 90 49
fax: +33 02 47 49 91 49
_____________________________________________
Metakit mailing list - Metakit@...
http://www.equi4.com/mailman/listinfo/metakit