chroot of iPlanet 6.0 and Siebel....

[Jonathan Katz]

All,

I'm attempting to establish a chrooted environment for an iPlanet
6.0SP9 webserver with Siebel web extensions installed for our
enterprise on Solaris 9. I've been able to operate the SSL-based
webserver in the chrooted environment with no major problems. However,
when I enable the Siebel web extensions the webserver fails to start;
I'm not even prompted for the SSL password upon start, the webserver
coredumps. Based on error messages it looks like I'm missing
libraries, but after examining LD_LIBRARY_PATH and judicious use of
ldd I'm coming up short...

The Siebel Web Extensions (SWE) are NSAPI C++ libraries added in
through some $LD_LIBRARY_PATH voodoo in the start script and through a
'load-modules' in the magnus.conf. If I remove the 'load-modules' line
in the magnus.conf for the SWE and the "Service method" line from
obj.conf the webserver doesn't reference the SWE and it operates just
fine.

My chroot is placed in /opt/sandbox. The apps all think they live in
/opt/app; my app installation was copied from a working server that
uses /opt/app. So the apps are physically located in
/opt/sandbox/opt/app. As a test, on the OS I symlinked
/opt/sandbox/opt/app to /opt/app. When I launch iPlanet with the SWE
from /opt/app without the chroot they work fine. Once within the
chroot it fails miserably. If I take the SWE out of the config iPlanet
will work within the chroot.

All of the application and its supporting libraries lives under
/opt/app/siebel77 and /opt/app/iplanet. For the chroot I copied over
/usr/lib, /usr/bin, /usr/java, /usr/j2se, /usr/java1.2,  /usr/platform
and /usr/share for good measure. lib and bin are symlinked from the
chroot to usr/bin and usr/lib ... /etc/ has passwd, hosts, group and
netconfig. I can't think of what else could be missing. All of the
required items for /dev exist, too and are created with the
appropriate major/minor numbers, ownership and permission due to some
pretty cool script parsing of /etc/minor_perm and ls -lL.

When running with the SWE enabled in the chroot the following error
shows in the iPlanet error log....

[20/May/2005:11:28:32] info (14969): [LS ls1] https://my_server, port
443 ready to accept requests
[20/May/2005:11:28:32] info (14969): A new configuration was
successfully installed
[20/May/2005:11:30:46] info (14992): successful server startup
[20/May/2005:11:30:46] info (14992):
iPlanet-WebServer-Enterprise/6.0SP9 B11/04/2004 06:35
[20/May/2005:11:30:46] catastrophe (14992): Server crash detected
(signal SIGSEGV)
[20/May/2005:11:30:46] info (14992): Crash occurred in NSAPI SAF swe-init
[20/May/2005:11:30:46] info (14992): Crash occurred in function
__1cLCcfLogEvent6FpnJsrdObjTag_1pkHrknMCCFFormatArg_66666_i_ from
module
  /opt/app/siebel77/bin/libsslcshar.so

When I use ldd against the file libsslcshar.so with the proper
LD_LIBRARY_PATH set (the one that is still in the iPlanet start file)
it resolves everything OK... the few libraries that are unresolved are
also unresolved outside of the chroot where the software functions as
it should. The paths and files all exist under the chroot as they'd be
seen in the chroot. I've run ldd against the libraries that
libsslcshar.so links against, too. They all seem to resolve OK and
exist in the proper places in the chroot. I'm really baffled by this.
No Siebel config files reference a pathname that does not exist under
the chroot, either.

I also ran lsof on the working iPlanet processes with the SWE outside
of the chroot. I found nothing listed that wasn't copied over into the
chroot.

I don't know what else could be broken. Does anyone have an idea if I
left any files out of the chroot environment? Has anyone established a
chrooted webserver environment with Siebel Web Extensions or other 3rd
party apps? If so, have you encountered the same pitfalls? What were
your workarounds? We will also open a call with Siebel but I expect
little assistance from them on the subject or a declaration that what
I'm trying to accomplish is unsupported by them. I did find some Sun
documentation that referenced a bug with chroot and servlets, but it
dates to 4.0SP3... See
http://docs.sun.com/source/816-5676-10/rn41sp7.html for details....
but I'm not running a JSP-style servlet.

Thanks in advance for any assistance... I know longer-term it would
make sense to go with a zone and Solaris 10 but that isn't an option
for us right now.

-Jon

[John Rowan Littell]

-----BEGIN PGP SIGNED MESSAGE-----

Lo, Jonathan Katz and the teakettle whistled in unison:

> All,
>
> I'm attempting to establish a chrooted environment for an iPlanet
> 6.0SP9 webserver with Siebel web extensions installed for our
> enterprise on Solaris 9. I've been able to operate the SSL-based
> webserver in the chrooted environment with no major problems. However,
> when I enable the Siebel web extensions the webserver fails to start;
> I'm not even prompted for the SSL password upon start, the webserver
> coredumps. Based on error messages it looks like I'm missing
> libraries, but after examining LD_LIBRARY_PATH and judicious use of
> ldd I'm coming up short...

[snip]

I'm afraid I can't help with the specifics of this one, but one
approach I've tried with some success when I've had really peculiar
run time problems is using truss to find out what system calls the
errant program is making and what it's getting out of them.  That
sometimes uncovers file open failures that I'd never find otherwise or
similar odd problems.

   --rowan

- --
John "Rowan" Littell
Systems Administrator
Earlham College Computing Services
http://www.earlham.edu/~littejo/
2005-05-24 20:12
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: http://www.earlham.edu/~littejo/littejo.asc

iQCVAwUBQpPSUJdUNSJ2nf/5AQGIOgP+NKfCGj5Jce8DML+EvRxc4zbNZuUtu/Y+
JuclBB3I5ulmFsnG8hm2SHOAZ2f52Ay1XLpUdQ6xEgpsl2OqYDkh+JcUxky3w03F
mq3yxMuL2egad50UWzOOG+3vJM+7kW7TGJBG57yxDNT32igT8psrnD7XQQpBiHQJ
jWrnO+Gs/yY=
=bV4+
-----END PGP SIGNATURE-----

[Warren Belfer]

>Lo, Jonathan Katz and the teakettle whistled in unison:
>
>> All,
>>
>> I'm attempting to establish a chrooted environment for an iPlanet
>> 6.0SP9 webserver with Siebel web extensions installed for our
>> enterprise on Solaris 9. I've been able to operate the SSL-based
>> webserver in the chrooted environment with no major problems. However,
>> when I enable the Siebel web extensions the webserver fails to start;
>> I'm not even prompted for the SSL password upon start, the webserver
>> coredumps. Based on error messages it looks like I'm missing
>> libraries, but after examining LD_LIBRARY_PATH and judicious use of
>> ldd I'm coming up short...
>[snip]
>
>I'm afraid I can't help with the specifics of this one, but one
>approach I've tried with some success when I've had really peculiar
>run time problems is using truss to find out what system calls the
>errant program is making and what it's getting out of them.  That
>sometimes uncovers file open failures that I'd never find otherwise or
>similar odd problems.

You might want to try 'pldd' on a system where the application is
running correctly (not in a chroot)  'pldd' will give you not only
the directly required libraries, like 'ldd' will, but it also lists
any librarires which those libraries call. That should give you all
of the dependencies (library-wise at least)

HTH,

Warren