Managing large python/c++ project

[malat]

Hi there,

  I am currently involved in managing a large python/c++ project
(using swig to automagically wrap c++ code to python). So far my
current experience was that python was a second class citizen and
extremely little python code was written and everything seemed to
work.

  Now this is the contrary, large portion of code are written python
with a few expection of c++ code for core algorithms. So as you might
have guess I am coming with a c++ background into the python world.

  Let's consider the following layout for project 'A':

  A/
    moduleA.py
    B/
      moduleB.py
    C/
      moduleC.py
      module1.cxx
      module1.h
      module1.i
  A-binary-module-gcc43/
    _module1.so
    module1.py

  I have the current questions (I guess it's all about convention):

Q1. How should I import module1 (a c++ module that is compiled/wrap
into python) ? Because I have multiple compilers, I have a binary
directory per compiler. My guess was that PYTHONPATH should contain
the toplevel project *source* directory and the toplevel project
*binary* directory.

Possible answers:
A1.  'import module1'...well in that case I should really make sure
that noone ever tries to create a directory names module1 in directory
A/ otherwise 'import module1' statement is changing meaning.

A2. 'import A-binary-module-gcc43.module1' ... well this is safer and
would avoid namespace collision but the python code becomes dependent
(implicitely) of a compiler.

A3. Leave the module1 where it belong : in A/B/C subdirectory ... in
this case I need to have a source directory per compiler, which means
I need either some subttle trick with symlinks (lndir) or be ready to
have out of sync projects.

Q2. How do I import moduleA & moduleB ? Those are a simple python
modules that are located in A/. Behavior should be symmetric.

A1.
  'from A import moduleA'
  'from A.B import moduleB'

A2.
  'from A import moduleA'
  'from B import moduleB'

A3.
  'import moduleA'
  'import moduleB'

Basically A1 to A3 are organized in complexity of PYTHONPATH. A1 is
great because importing a moduleD in the future will just works
(PYTHONPATH is unchanged). A2 & A3 make the layout of the project
impact on the PYTHONPATH.

Thanks
-Mathieu

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user

[Brian Cole-3]

We ship C++, Python, and Java libraries. We don't write much library  
code in Python so our needs may be a little different. We've recently  
redesigned our distributed layout to alleviate the need to set  
LD_LIBRARY_PATH. So the design looks like:

toplevel/
    moduleA.py <-- auto generated from SWIG
    moduleB.py <-- auto generated from SWIG
    libs/
      __init__.py <-- some magic incantations in here
      operating-system_compiler_arch_python-version/
        _moduleA.so
        libmoduleA.so
        _moduleB.so
        libmoduleB.so
        libmutualdependency.so

The __init__.py script contains a function like this:
def GetModule(name):
     args = imp.find_module(name, ["operating-
system_compiler_arch_python-version"])
     return imp.load_module(name, *args)

We already post-process the SWIG generated .py files anyway so it was  
trivial to replace:
import _moduleA
With:
from libs import GetModule
_moduleA = GetModule("_moduleA")

The logic inside GetModule is obviously a little more complex to parse  
out all the necessary system information on the fly. This allows users  
to unpack multiple platform/compiler/python versions into the same  
directory structure (usually an NFS mount), set PYTHONPATH once, and  
__init__.py inside libs figures out the correct module to import.

There is one more shared library trick to make this work, using ORIGIN  
with the --rpath argument to the linker. (http://linuxreviews.org/man/ld.so/#lbAE 
) This allows _moduleA.so be able to find libmoduleA.so inside the  
same directory as itself, no matter where it got untar'd. We've also  
used something similar to this with Java across a lot of target  
platforms, so we're fairly certain about it's portability.

So users import our library using the following idioms:
from toplevel.moduleA import *
from toplevel import moduleA
...etc

The 'toplevel' directory prevents our module names from polluting  
PYTHONPATH. However, we've never tried tightly coupling full blown  
python modules into this structure, so your mileage may vary.

-Brian

On Jun 21, 2008, at 3:10 AM, mathieu wrote:

>
> Hi there,
>
>  I am currently involved in managing a large python/c++ project
> (using swig to automagically wrap c++ code to python). So far my
> current experience was that python was a second class citizen and
> extremely little python code was written and everything seemed to
> work.
>
>  Now this is the contrary, large portion of code are written python
> with a few expection of c++ code for core algorithms. So as you might
> have guess I am coming with a c++ background into the python world.
>
>  Let's consider the following layout for project 'A':
>
>  A/
>    moduleA.py
>    B/
>      moduleB.py
>    C/
>      moduleC.py
>      module1.cxx
>      module1.h
>      module1.i
>  A-binary-module-gcc43/
>    _module1.so
>    module1.py
>
>  I have the current questions (I guess it's all about convention):
>
> Q1. How should I import module1 (a c++ module that is compiled/wrap
> into python) ? Because I have multiple compilers, I have a binary
> directory per compiler. My guess was that PYTHONPATH should contain
> the toplevel project *source* directory and the toplevel project
> *binary* directory.
>
> Possible answers:
> A1.  'import module1'...well in that case I should really make sure
> that noone ever tries to create a directory names module1 in directory
> A/ otherwise 'import module1' statement is changing meaning.
>
> A2. 'import A-binary-module-gcc43.module1' ... well this is safer and
> would avoid namespace collision but the python code becomes dependent
> (implicitely) of a compiler.
>
> A3. Leave the module1 where it belong : in A/B/C subdirectory ... in
> this case I need to have a source directory per compiler, which means
> I need either some subttle trick with symlinks (lndir) or be ready to
> have out of sync projects.
>
> Q2. How do I import moduleA & moduleB ? Those are a simple python
> modules that are located in A/. Behavior should be symmetric.
>
> A1.
>  'from A import moduleA'
>  'from A.B import moduleB'
>
> A2.
>  'from A import moduleA'
>  'from B import moduleB'
>
> A3.
>  'import moduleA'
>  'import moduleB'
>
> Basically A1 to A3 are organized in complexity of PYTHONPATH. A1 is
> great because importing a moduleD in the future will just works
> (PYTHONPATH is unchanged). A2 & A3 make the layout of the project
> impact on the PYTHONPATH.
>
> Thanks
> -Mathieu
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Swig-user mailing list
Swig-user@...
https://lists.sourceforge.net/lists/listinfo/swig-user