Broadening the scope of the negative connection cache

[Marc VanHeyningen]

I've encountered some minor performance in Samba that come up when a PDC
is down.  Briefly, the ability of the negative connection cache to limit
the time wasted attempting to contact that PDC is limited by the fact
that it's implemented as a linked list and limited in scope to the
process where the failed connection was performed.

 

The obvious solution would be to migrate the negative connection cache
to be stored in a different way, probably as a TDB file.  This would
allow every child process to share the failure information.

 

I guess the questions are: Does this make sense?  If so, should this be
a wholesale replacement or conditional on something like #ifdef
CONNCACHE_USING_TDB?  I can't really imagine that the performance
considerations would be so great that anyone would want to revert to the
old way.

 

Thanks!

Marc

 

[Gerald (Jerry) Carter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marc,

> I've encountered some minor performance in Samba that come
> up when a PDC is down.  Briefly, the ability of the negative
> connection cache to limit the time wasted attempting to
> contact that PDC is limited by the fact that it's implemented
> as a linked list and limited in scope to the process
> where the failed connection was performed.
>
> The obvious solution would be to migrate the negative
> connection cache to be stored in a different way, probably
> as a TDB file.  This would allow every child process to share
> the failure information.

I believe the easiest way to achieve this is simply to
use the gencache API and set a 60 second TTL on entries.




cheers, jerry
- --
=====================================================================
Samba                                    ------- http://www.samba.org
Likewise Software          ---------  http://www.likewisesoftware.com
"What man is a man who does not make the world better?"      --Balian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKaXXIR7qMdg1EfYRAlCDAJ9ksY8k21gzjbr1N3JcpzN4SN4EFACfZfK6
X4l4N90Q6W5mQq9BrTsohBg=
=/Q46
-----END PGP SIGNATURE-----

[Volker Lendecke]

On Tue, May 13, 2008 at 09:29:43AM -0500, Gerald (Jerry) Carter wrote:
> > The obvious solution would be to migrate the negative
> > connection cache to be stored in a different way, probably
> > as a TDB file.  This would allow every child process to share
> > the failure information.
>
> I believe the easiest way to achieve this is simply to
> use the gencache API and set a 60 second TTL on entries.

While there -- can we move the idmap cache there as well?

What that would do: smbd could also put stuff in there. This
is really necessary for people having ACLs and "hide
unreadable". For each file we have to translate the gids to
sids. This basically took down a PDC's LDAP server of a
customer of mine. No winbind around, just smbd.

Volker

[Gerald (Jerry) Carter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Volker Lendecke wrote:

> While there -- can we move the idmap cache there as well?
>
> What that would do: smbd could also put stuff in there. This
> is really necessary for people having ACLs and "hide
> unreadable". For each file we have to translate the gids to
> sids. This basically took down a PDC's LDAP server of a
> customer of mine. No winbind around, just smbd.

Hmm....not sure I'm initially as supportive of that idea.  But I'll
think on it and maybe convince myself differently.

Why can't they run winbindd?  Internally smbd has a sid/uid/gid
cache.  Maybe that should be in gencache.  But I'd like to keep
Winbindd's idmap cache separate.

PS: I do believe that Winbind's idmap cache needs to be cleaned
up.  For example, caching the forward and reverse map entries
should be in a single transaction.





cheers, jerry
- --
=====================================================================
Samba                                    ------- http://www.samba.org
Likewise Software          ---------  http://www.likewisesoftware.com
"What man is a man who does not make the world better?"      --Balian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKakJIR7qMdg1EfYRAn9OAKDmTMFFlh/mpwpG/1oInD07QmsycwCgsmYX
mNYLgJhemYq/rPVB0FnACoU=
=huP5
-----END PGP SIGNATURE-----

[Volker Lendecke]

On Tue, May 13, 2008 at 09:43:21AM -0500, Gerald (Jerry) Carter wrote:
> Why can't they run winbindd?  Internally smbd has a sid/uid/gid
> cache.  Maybe that should be in gencache.  But I'd like to keep
> Winbindd's idmap cache separate.

Hmmm. Have to revisit gid2sid then. I thought that for "our"
SAM smbd on a PDC does the mapping without asking winbind. I
might be wrong.

Sorry for the noise.

Volker

[Gerald (Jerry) Carter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Volker Lendecke wrote:
> On Tue, May 13, 2008 at 09:43:21AM -0500, Gerald (Jerry) Carter wrote:
>> Why can't they run winbindd?  Internally smbd has a sid/uid/gid
>> cache.  Maybe that should be in gencache.  But I'd like to keep
>> Winbindd's idmap cache separate.
>
> Hmmm. Have to revisit gid2sid then. I thought that for "our"
> SAM smbd on a PDC does the mapping without asking winbind. I
> might be wrong.

No. Your right.  I missed the fact that it was a PDC.  I'm happy
if we convert the uid/gid cache in smbd from memcache to gencache.
That seems like a logical thing to do.





cheers, jerry
- --
=====================================================================
Samba                                    ------- http://www.samba.org
Likewise Software          ---------  http://www.likewisesoftware.com
"What man is a man who does not make the world better?"      --Balian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKawHIR7qMdg1EfYRAjd6AKDShsFNiTM1YyzRRDPvGu1cNk8UIACdFfoN
nAz1E4y3pMrsHq6XfdRoDns=
=rQw2
-----END PGP SIGNATURE-----

[simo-7]

On Tue, 2008-05-13 at 09:43 -0500, Gerald (Jerry) Carter wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Volker Lendecke wrote:
>
> > While there -- can we move the idmap cache there as well?
> >
> > What that would do: smbd could also put stuff in there. This
> > is really necessary for people having ACLs and "hide
> > unreadable". For each file we have to translate the gids to
> > sids. This basically took down a PDC's LDAP server of a
> > customer of mine. No winbind around, just smbd.
>
> Hmm....not sure I'm initially as supportive of that idea.  But I'll
> think on it and maybe convince myself differently.
>
> Why can't they run winbindd?  Internally smbd has a sid/uid/gid
> cache.  Maybe that should be in gencache.  But I'd like to keep
> Winbindd's idmap cache separate.
>
> PS: I do believe that Winbind's idmap cache needs to be cleaned
> up.  For example, caching the forward and reverse map entries
> should be in a single transaction.

We should also make the cache be readable by any winbindd process too,
so that we save some round trips, it was not done in initial design to
keep it simpler, but it is time to IMO.

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <simo@...>
Senior Software Engineer at Red Hat Inc. <ssorce@...>

[Gerald (Jerry) Carter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

simo wrote:

>> PS: I do believe that Winbind's idmap cache needs to be cleaned
>> up.  For example, caching the forward and reverse map entries
>> should be in a single transaction.
>
> We should also make the cache be readable by any winbindd process too,
> so that we save some round trips, it was not done in initial design to
> keep it simpler, but it is time to IMO.

That's a different issue.  In general the design should be
that tha parent winbindd process always hits cache first
and only calls out to a child process when the cache is
empty or expired.  This is more in line with the original
dual-daemon design Tridge implemented.  Which I think
would work really well with the state machine model we
have now.

So it's not just the idmap cache that needs to be readable,
but the entire cache manager should be moved to the parent.





cheers, jerry

- --
=====================================================================
Samba                                    ------- http://www.samba.org
Likewise Software          ---------  http://www.likewisesoftware.com
"What man is a man who does not make the world better?"      --Balian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKa6GIR7qMdg1EfYRAveYAKCNjGfr0rV2j7V3tf/R05e3tLJskQCgw4gB
YUxD8C59XX4ABPVCJqvMXrw=
=9FRJ
-----END PGP SIGNATURE-----

[simo-7]

On Tue, 2008-05-13 at 10:06 -0500, Gerald (Jerry) Carter wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> simo wrote:
>
> >> PS: I do believe that Winbind's idmap cache needs to be cleaned
> >> up.  For example, caching the forward and reverse map entries
> >> should be in a single transaction.
> >
> > We should also make the cache be readable by any winbindd process too,
> > so that we save some round trips, it was not done in initial design to
> > keep it simpler, but it is time to IMO.
>
> That's a different issue.  In general the design should be
> that tha parent winbindd process always hits cache first
> and only calls out to a child process when the cache is
> empty or expired.  This is more in line with the original
> dual-daemon design Tridge implemented.  Which I think
> would work really well with the state machine model we
> have now.
>
> So it's not just the idmap cache that needs to be readable,
> but the entire cache manager should be moved to the parent.

That's what I meant.

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <simo@...>
Senior Software Engineer at Red Hat Inc. <ssorce@...>

[Marc VanHeyningen]

Gerald (Jerry) Carter sed:
>> The obvious solution would be to migrate the negative
>> connection cache to be stored in a different way, probably
>> as a TDB file.  This would allow every child process to share
>> the failure information.
>
>I believe the easiest way to achieve this is simply to
>use the gencache API and set a 60 second TTL on entries.

Sounds good, but a 60 second TTL seems too short to me.  Attempting
to find a host and having it fail to respond can take a significant
fraction of 60 seconds in some (admittedly unusual) cases.

Probably the best system is an adaptive one, which increases the
TTL of the negative entry the longer the target remains inaccessible,
but that's more complexity than this situation really warrants.

Marc

[Gerald (Jerry) Carter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marc VanHeyningen wrote:

> Gerald (Jerry) Carter sed:
>>> The obvious solution would be to migrate the negative
>>> connection cache to be stored in a different way, probably
>>> as a TDB file.  This would allow every child process to share
>>> the failure information.
>> I believe the easiest way to achieve this is simply to
>> use the gencache API and set a 60 second TTL on entries.
>
> Sounds good, but a 60 second TTL seems too short to me.  Attempting
> to find a host and having it fail to respond can take a significant
> fraction of 60 seconds in some (admittedly unusual) cases.
>
> Probably the best system is an adaptive one, which increases the
> TTL of the negative entry the longer the target remains inaccessible,
> but that's more complexity than this situation really warrants.

Anything to reduce timeouts is fine.  Let's just make one
change at a time.  Moving to gencache here is the first step.



jerry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKyVtIR7qMdg1EfYRAtovAJ45F8Ym/m7Y7OmYDeeoFtOXmhPd/gCff6VL
NVWMfc/rfPpblWJkXozYfPM=
=zU1j
-----END PGP SIGNATURE-----

[Sam Liddicott]

I think the timeout should relate to the amount of time that will be wasted to timeout again and the exponential decay of how much time has (recently) been wasted.

Iptables connrate match has a good formula.

Sam



-----Original Message-----
From: Gerald (Jerry) Carter <jerry@...>
Sent: 14 May 2008 18:46
To: Marc VanHeyningen <marc.vanheyningen@...>
Cc: samba-technical@...
Subject: Re: Broadening the scope of the negative connection cache

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marc VanHeyningen wrote:

> Gerald (Jerry) Carter sed:
>>> The obvious solution would be to migrate the negative
>>> connection cache to be stored in a different way, probably
>>> as a TDB file.  This would allow every child process to share
>>> the failure information.
>> I believe the easiest way to achieve this is simply to
>> use the gencache API and set a 60 second TTL on entries.
>
> Sounds good, but a 60 second TTL seems too short to me.  Attempting
> to find a host and having it fail to respond can take a significant
> fraction of 60 seconds in some (admittedly unusual) cases.
>
> Probably the best system is an adaptive one, which increases the
> TTL of the negative entry the longer the target remains inaccessible,
> but that's more complexity than this situation really warrants.

Anything to reduce timeouts is fine.  Let's just make one
change at a time.  Moving to gencache here is the first step.



jerry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKyVtIR7qMdg1EfYRAtovAJ45F8Ym/m7Y7OmYDeeoFtOXmhPd/gCff6VL
NVWMfc/rfPpblWJkXozYfPM=
=zU1j
-----END PGP SIGNATURE-----