Bug#447609: ldconfig triggerisation

[Ian Jackson-3]

Source: glibc
Version: 2.6.1-6
Severity: wishlist
Tags: patch

The attached patch triggerises the invocation of ldconfig by package
maintainer scripts.

By `triggerises' I mean that the patch arranges for ldconfig
invocations by maintainer scripts to call dpkg-trigger instead of
ldconfig.  ldconfig will be actually run out of glibc's maintainer
script during trigger processing.  The consequence is that all of the
ldconfig invocations during a dpkg run are deferred, and instead
ldconfig is run once at the end.

The understanding on which we base this approach is that after library
installation (which is when ldconfig is used in maintainer scripts) it
is always safe to defer the execution of ldconfig.  Ie, that after a
new library has been installed or an existing library upgraded,
programs which link against the library will work even though ldconfig
hasn't been run.  We understand that not running ldconfig will incur
some performance penalty during the upgrade process but in practice
this is far outweighed by the cost of repeatedly running ldconfig.

We took the approach of renaming ldconfig to ldconfig.real and
replacing it with a wrapper script.  This is unfortunately necessary
because maintainer scripts are in the habit of calling ldconfig
directly.  An alternative approach would be to change all of the
packages not to call ldconfig but instead to call a new script but
this would involve a much more complicated and lengthy transition.

The patch is safe to use with a non-triggers-supporting dpkg and in
all transitional states: where the trigger system is not properly set
up yet, ldconfig is run as normal.  Note that sid's dpkg does not yet
have the triggers patch merged but there has been extensive discussion
of the design and interfaces for triggers and the API should IMO be
considered stable.

These changes have been tested and released as part of Ubuntu 7.10 aka
`gutsy gibbon'.  The patch below is the consilidation of the results
of our testing.

So we believe that this patch can and should be safely applied to
sid's glibc straight away.

Ian.


diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/debhelper.in/libc.postinst glibc-2.6.1/debian/debhelper.in/libc.postinst
--- orig/glibc-2.6.1/debian/debhelper.in/libc.postinst 2007-10-22 15:40:11.000000000 +0100
+++ glibc-2.6.1/debian/debhelper.in/libc.postinst 2007-10-22 15:38:11.000000000 +0100
@@ -5,6 +5,15 @@
 type=$1
 preversion=$2
 
+if [ "x$type" = xtriggered ]
+then
+ LDCONFIG_NOTRIGGER=y
+ export LDCONFIG_NOTRIGGER
+ echo ldconfig deferred processing now taking place
+ ldconfig
+ exit 0
+fi
+
 package_name()
 {
     echo LIBC
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/debhelper.in/libc.triggers glibc-2.6.1/debian/debhelper.in/libc.triggers
--- orig/glibc-2.6.1/debian/debhelper.in/libc.triggers 1970-01-01 01:00:00.000000000 +0100
+++ glibc-2.6.1/debian/debhelper.in/libc.triggers 2007-10-22 15:38:11.000000000 +0100
@@ -0,0 +1 @@
+interest ldconfig
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/local/ldconfig_wrap glibc-2.6.1/debian/local/ldconfig_wrap
--- orig/glibc-2.6.1/debian/local/ldconfig_wrap 1970-01-01 01:00:00.000000000 +0100
+++ glibc-2.6.1/debian/local/ldconfig_wrap 2007-10-22 15:39:01.000000000 +0100
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+if  test $# = 0 \
+    && test x"$LDCONFIG_NOTRIGGER" = x \
+ && test x"$DPKG_MAINTSCRIPT_PACKAGE" != x \
+ && dpkg-trigger --check-supported 2>/dev/null \
+ && dpkg --compare-versions "$DPKG_RUNNING_VERSION" ge '1.14.5ubuntu10~~'
+then
+ if dpkg-trigger --no-await ldconfig; then
+ if test x"$LDCONFIG_TRIGGER_DEBUG" != x; then
+ echo "ldconfig: wrapper deferring update (trigger activated)"
+ fi
+ exit 0
+ fi
+fi
+
+exec /sbin/ldconfig.real "$@"
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/rules.d/debhelper.mk glibc-2.6.1/debian/rules.d/debhelper.mk
--- orig/glibc-2.6.1/debian/rules.d/debhelper.mk 2007-10-22 15:40:11.000000000 +0100
+++ glibc-2.6.1/debian/rules.d/debhelper.mk 2007-10-22 15:38:11.000000000 +0100
@@ -59,6 +59,13 @@
     dh_install -p$(curpass) debian/bug/$(curpass) usr/share/bug; \
  fi
 
+ set -ex; case $(curpass) in libc6|libc6.1) \
+ mv debian/$(curpass)/sbin/ldconfig \
+ debian/$(curpass)/sbin/ldconfig.real; \
+ install -m755 -o0 -g0 debian/local/ldconfig_wrap \
+ debian/$(curpass)/sbin/ldconfig; \
+ ;; esac
+
  # extra_debhelper_pkg_install is used for debhelper.mk only.
  # when you want to install extra packages, use extra_pkg_install.
  $(call xx,extra_debhelper_pkg_install)
@@ -118,6 +125,11 @@
  debian/$(curpass)/usr/share/lintian/overrides/$(curpass) ; \
  fi
 
+ if [ -f debian/$(curpass).triggers ] ; then \
+ install -m 644 -o root -g root debian/$(curpass).triggers \
+ debian/$(curpass)/DEBIAN/triggers ; \
+ fi
+
  if [ -f debian/$(curpass).linda ] ; then \
  install -d -m 755 -o root -g root debian/$(curpass)/usr/share/linda/overrides/ ; \
  install -m 644 -o root -g root debian/$(curpass).linda \

[Daniel Jacobowitz-2]

On Mon, Oct 22, 2007 at 04:07:05PM +0100, Ian Jackson wrote:
> The understanding on which we base this approach is that after library
> installation (which is when ldconfig is used in maintainer scripts) it
> is always safe to defer the execution of ldconfig.  Ie, that after a
> new library has been installed or an existing library upgraded,
> programs which link against the library will work even though ldconfig
> hasn't been run.  We understand that not running ldconfig will incur
> some performance penalty during the upgrade process but in practice
> this is far outweighed by the cost of repeatedly running ldconfig.

Note that this is usually true but not always; it may be true
enough for our purposes but I want to set the record straight.

ld.so searches LD_LIBRARY_PATH plus /usr/lib and /lib, and standard
hwcap subdirectories of those.  It does not read /etc/ld.so.conf, nor
does it search the "tls" subdirectories still used in some places to
find NPTL.

The only failure case I can think of would be a package which places
libraries in the multi-arch directories, which Debian locates using a
file in /etc/ld.so.conf.d, and the same or another package which runs
a newly installed program using the library from the first package
in its postinst.

If usage of those directories is planned to increase this may become
a problem.

--
Daniel Jacobowitz
CodeSourcery



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Ian Jackson]

Daniel Jacobowitz writes ("Re: Bug#447609: ldconfig triggerisation"):
> On Mon, Oct 22, 2007 at 04:07:05PM +0100, Ian Jackson wrote:
> > [assumptions]
>
> Note that this is usually true but not always; it may be true
> enough for our purposes but I want to set the record straight.

Thanks.

> The only failure case I can think of would be a package which places
> libraries in the multi-arch directories, which Debian locates using a
> file in /etc/ld.so.conf.d, and the same or another package which runs
> a newly installed program using the library from the first package
> in its postinst.

Do such packages typically invoke ldconfig in the same way as a normal
package does ?  If not then we could distinguish the two kinds of
ldconfig call.

> If usage of those directories is planned to increase this may become
> a problem.

Yes, I see.

Joey Hess writes ("Re: Bug#447609: ldconfig triggerisation"):
> Couldn't file triggers be used, so ldconfig is triggered after any
> package installs a library file? That's much more how I expected
> triggers to be used, rather than needing an ugly ldconfig wrapper. I
> think it also addresses drow's point about libraries in nonstandard
> locations, since those packages could just run ldconfig as usual.
> Meanwhile, packages installing libraries to standard locations could
> stop calling ldconfig.

Unfortunately because of the way /lib and particularly /usr/lib are
dumping grounds, file triggers wouldn't work properly.  A file trigger
is activated by packages' files whose pathnames start with the name of
the trigger.  So the trigger would have to be /usr/lib/ but that
covers enormous amounts of junk.

Also, doing it like that would mean we should probably wait with
starting on those changes to packages until the dpkg (and apt and
aptitude) with triggers were actually deployed, as otherwise in the
meantime ldconfig wouldn't run at all.

Ian.



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Daniel Jacobowitz-2]

On Tue, Oct 23, 2007 at 11:11:37AM +0100, Ian Jackson wrote:
> > The only failure case I can think of would be a package which places
> > libraries in the multi-arch directories, which Debian locates using a
> > file in /etc/ld.so.conf.d, and the same or another package which runs
> > a newly installed program using the library from the first package
> > in its postinst.
>
> Do such packages typically invoke ldconfig in the same way as a normal
> package does ?  If not then we could distinguish the two kinds of
> ldconfig call.

I don't think there would be any difference.

--
Daniel Jacobowitz
CodeSourcery



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Clint Adams]

On Tue, Oct 23, 2007 at 11:11:37AM +0100, Ian Jackson wrote:
> Unfortunately because of the way /lib and particularly /usr/lib are
> dumping grounds, file triggers wouldn't work properly.  A file trigger
> is activated by packages' files whose pathnames start with the name of
> the trigger.  So the trigger would have to be /usr/lib/ but that
> covers enormous amounts of junk.

Why not just move to use of dpkg-trigger piecemeal in each postinst?
That would involve no more invocations of ldconfig than we are already
enduring.



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Ian Jackson]

Daniel Jacobowitz writes ("Re: Bug#447609: ldconfig triggerisation"):

> On Tue, Oct 23, 2007 at 11:11:37AM +0100, Ian Jackson wrote:
> > > The only failure case I can think of would be a package which places
> > > libraries in the multi-arch directories, which Debian locates using a
> > > file in /etc/ld.so.conf.d, and the same or another package which runs
> > > a newly installed program using the library from the first package
> > > in its postinst.
> >
> > Do such packages typically invoke ldconfig in the same way as a normal
> > package does ?  If not then we could distinguish the two kinds of
> > ldconfig call.
>
> I don't think there would be any difference.

Hrm.  That's troublesome.  Can we insist on them making such a
different invocation ?

If we can't then the ldconfig provided by glibc can't tell the
difference and AFAICS that means we can't provide the optimisation
without updating all packages.

Ian.



--
To UNSUBSCRIBE, email to debian-glibc-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Ian Jackson]

Clint Adams writes ("Re: Bug#447609: ldconfig triggerisation"):
> Why not just move to use of dpkg-trigger piecemeal in each postinst?
> That would involve no more invocations of ldconfig than we are already
> enduring.

Because that would involve a great deal of additional dependency
complexity: each such package would have to depend on the
triggers-capable dpkg and the triggers-interested libc.

Ian.



--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Debian Bug Tracking System]

Your message dated Tue, 22 Jul 2008 15:14:52 +0200
with message-id <20080722131452.GA4340@...>
and subject line Re: Bug#447609: ldconfig triggerisation
has caused the Debian Bug report #447609,
regarding ldconfig triggerisation
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@...
immediately.)


--
447609: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=447609
Debian Bug Tracking System
Contact owner@... with problems

Source: glibc
Version: 2.6.1-6
Severity: wishlist
Tags: patch

The attached patch triggerises the invocation of ldconfig by package
maintainer scripts.

By `triggerises' I mean that the patch arranges for ldconfig
invocations by maintainer scripts to call dpkg-trigger instead of
ldconfig.  ldconfig will be actually run out of glibc's maintainer
script during trigger processing.  The consequence is that all of the
ldconfig invocations during a dpkg run are deferred, and instead
ldconfig is run once at the end.

The understanding on which we base this approach is that after library
installation (which is when ldconfig is used in maintainer scripts) it
is always safe to defer the execution of ldconfig.  Ie, that after a
new library has been installed or an existing library upgraded,
programs which link against the library will work even though ldconfig
hasn't been run.  We understand that not running ldconfig will incur
some performance penalty during the upgrade process but in practice
this is far outweighed by the cost of repeatedly running ldconfig.

We took the approach of renaming ldconfig to ldconfig.real and
replacing it with a wrapper script.  This is unfortunately necessary
because maintainer scripts are in the habit of calling ldconfig
directly.  An alternative approach would be to change all of the
packages not to call ldconfig but instead to call a new script but
this would involve a much more complicated and lengthy transition.

The patch is safe to use with a non-triggers-supporting dpkg and in
all transitional states: where the trigger system is not properly set
up yet, ldconfig is run as normal.  Note that sid's dpkg does not yet
have the triggers patch merged but there has been extensive discussion
of the design and interfaces for triggers and the API should IMO be
considered stable.

These changes have been tested and released as part of Ubuntu 7.10 aka
`gutsy gibbon'.  The patch below is the consilidation of the results
of our testing.

So we believe that this patch can and should be safely applied to
sid's glibc straight away.

Ian.


diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/debhelper.in/libc.postinst glibc-2.6.1/debian/debhelper.in/libc.postinst
--- orig/glibc-2.6.1/debian/debhelper.in/libc.postinst 2007-10-22 15:40:11.000000000 +0100
+++ glibc-2.6.1/debian/debhelper.in/libc.postinst 2007-10-22 15:38:11.000000000 +0100
@@ -5,6 +5,15 @@
 type=$1
 preversion=$2
 
+if [ "x$type" = xtriggered ]
+then
+ LDCONFIG_NOTRIGGER=y
+ export LDCONFIG_NOTRIGGER
+ echo ldconfig deferred processing now taking place
+ ldconfig
+ exit 0
+fi
+
 package_name()
 {
     echo LIBC
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/debhelper.in/libc.triggers glibc-2.6.1/debian/debhelper.in/libc.triggers
--- orig/glibc-2.6.1/debian/debhelper.in/libc.triggers 1970-01-01 01:00:00.000000000 +0100
+++ glibc-2.6.1/debian/debhelper.in/libc.triggers 2007-10-22 15:38:11.000000000 +0100
@@ -0,0 +1 @@
+interest ldconfig
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/local/ldconfig_wrap glibc-2.6.1/debian/local/ldconfig_wrap
--- orig/glibc-2.6.1/debian/local/ldconfig_wrap 1970-01-01 01:00:00.000000000 +0100
+++ glibc-2.6.1/debian/local/ldconfig_wrap 2007-10-22 15:39:01.000000000 +0100
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+if  test $# = 0 \
+    && test x"$LDCONFIG_NOTRIGGER" = x \
+ && test x"$DPKG_MAINTSCRIPT_PACKAGE" != x \
+ && dpkg-trigger --check-supported 2>/dev/null \
+ && dpkg --compare-versions "$DPKG_RUNNING_VERSION" ge '1.14.5ubuntu10~~'
+then
+ if dpkg-trigger --no-await ldconfig; then
+ if test x"$LDCONFIG_TRIGGER_DEBUG" != x; then
+ echo "ldconfig: wrapper deferring update (trigger activated)"
+ fi
+ exit 0
+ fi
+fi
+
+exec /sbin/ldconfig.real "$@"
diff --exclude='*.orig' -ruN orig/glibc-2.6.1/debian/rules.d/debhelper.mk glibc-2.6.1/debian/rules.d/debhelper.mk
--- orig/glibc-2.6.1/debian/rules.d/debhelper.mk 2007-10-22 15:40:11.000000000 +0100
+++ glibc-2.6.1/debian/rules.d/debhelper.mk 2007-10-22 15:38:11.000000000 +0100
@@ -59,6 +59,13 @@
     dh_install -p$(curpass) debian/bug/$(curpass) usr/share/bug; \
  fi
 
+ set -ex; case $(curpass) in libc6|libc6.1) \
+ mv debian/$(curpass)/sbin/ldconfig \
+ debian/$(curpass)/sbin/ldconfig.real; \
+ install -m755 -o0 -g0 debian/local/ldconfig_wrap \
+ debian/$(curpass)/sbin/ldconfig; \
+ ;; esac
+
  # extra_debhelper_pkg_install is used for debhelper.mk only.
  # when you want to install extra packages, use extra_pkg_install.
  $(call xx,extra_debhelper_pkg_install)
@@ -118,6 +125,11 @@
  debian/$(curpass)/usr/share/lintian/overrides/$(curpass) ; \
  fi
 
+ if [ -f debian/$(curpass).triggers ] ; then \
+ install -m 644 -o root -g root debian/$(curpass).triggers \
+ debian/$(curpass)/DEBIAN/triggers ; \
+ fi
+
  if [ -f debian/$(curpass).linda ] ; then \
  install -d -m 755 -o root -g root debian/$(curpass)/usr/share/linda/overrides/ ; \
  install -m 644 -o root -g root debian/$(curpass).linda \

Version: 2.7-1

Since version 2.7-1, ldconfig includes a second level cache, which makes
calls to ldconfig very cheap, and only looks at the newer libraries.

I see that as another way to improve the speed of packages
installation, and more safer than ldconfig triggerisation.

I am therefore closing this bug.

--
  .''`.  Aurelien Jarno            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32@...         | aurelien@...
   `-    people.debian.org/~aurel32 | www.aurel32.net