HOWTO: Build DB XML 2.0 with Python Bindings on OS X

Sleepycat Software recently released Berkeley DB XML 2.0, a native XML database built on top of Berkley DB. The major change from DB XML 1 to 2 is the addition of XPath 2. and XQuery 1.0 support. Here’s what I did to build and install DB XML 2.0 along with python bindings on OS X 10.3.7.

Having previously built DB XML 1.0 on OS X, I was pleased to see that all of the prerequisite libraries (Xerces-C, Pathan 2, Berkeley DB) are now included in the DB XML source package. After downloading the tarball from the website (see link above), extracting it and building everything was easy: %> cd /Users/jclark/ext %> tar -xzf dbxml-2.0.9.tar.gz %> cd dbxml-2.0.9 %> sh buildall.sh

The buildall.sh script creates an install directory in the build root and installs all of the packages to this location. This gave me the following dir structure:

  Users
   + jclark
      + ext
         + dbxml-2.0.9
            + install
               + bin
               + docs
               + include
               + lib

Next I wanted to build python support, which is not built by buildall.sh. The README cautions that the python bindings require the pybsddb package, which ships with python 2.3 and later, but which must be compiled against the same version of Berkeley DB as XML DB (the XML DB package builds BDB 4.3.0 from source). OS X 10.3 comes with python 2.3 installed, but the default copy of bsddb doesn’t seem to work. I downloaded the pybsddb for BDB 4.3 from Sourceforge (via link above), and built it. Because of the nonstandard install location used by buildall.sh, I had to pass an extra param (--berkeley-db) to setup.py. :

  %> cd /Users/jclark/ext
  %> tar -xzf bsddb3-4.3.0
  %> cd bsddb3-4.3.0
  %> python setup.py --berkeley-db=/Users/jclark/ext/dbxml-2.0.9/install/ build
  %> python test.py
  %> python setup.py install

With a working bsddb, It was time to build XML DB’s python bindings. As mentioned in the main README and in the dbxml/src/python README, the custom BDB/DBXML/etc install location requires an edit to dbxml/src/python/setup.py. I adjusted the paths thusly:

  if os.name == "posix":
    db_home = "/Users/jclark/ext/dbxml-2.0.9/install"
    xerces_home = db_home
    pathan_home = db_home
    xquery_home = db_home

Building and installing was straightforward:

  %> cd /Users/jclark/ext/dbxml-2.0.9/src/python
  %> python setup.py build
  %> python setup.py install

Time to test:

  %> cd /Users/jclark/ext/dbxml-2.0.9/dbxml/examples/python
  %> python examples.py test
  Traceback (most recent call last):
    File "examples.py", line 6, in ?
      from bsddb.db import *
    File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
        python2.3/bsddb/__init__.py", line 40, in ?
      import _bsddb
  ImportError: No module named _bsddb

Uh-oh. However, a quick look inside examples.py reveals the solution. The import from bsddb.db applies when using the version of pybsddb that ships with python; when using a custom pybsddb build you must import from bsddb3. This involved commenting out one line and uncommenting another in examples.py. Let’s try again:

  %> python examples.py test
  Traceback (most recent call last):
    File "examples.py", line 7, in ?
      from dbxml import *
    File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/
        python2.3/site-packages/dbxml.py", line 5, in ?
      import _dbxml
  ImportError: Failure linking new module: libxerces-c.26.dylib: dyld: 
        python can't open library: libxerces-c.26.dylib  
        (No such file or directory, errno = 2)

Still no joy in Mudville. It’s at this point that I began to have trouble with how to proceed; my knowledge of *nix linking, loading, and such is limited. My rough understanding of dylibs is that they are dynamically linked libraries, a bit like Windows DLLs. The errors above (a couple of newlines were added for formatting) seem to indicate that the xerces-c lib can’t be located. I checked the output from building dbxml.py and confirmed that the correct (to my untrained eye) -l and -L flags were used to locate all of the required libs from /Users/jclark/ext/dbxml-2.0.9/install/libs.

My problem was, I didn’t really understand the dyld process. It was unclear to me whether dynamically linked libraries have the library location linked into the executable (as I suspected, but which didn’t explain the error) or if they rely on a search path (which probably defaults to /usr/lib or somesuch). In the end, man dyld proved somewhat enlightening. It turns out that there’s a slew of environment variables which can influence the locations searched for dylibs. It also turns out that otool -L can reveal information about linked libraries. Let’s see what we can learn (the two-step cd avoids a linebreak):

  %> cd /System/Library/Frameworks/Python.framework/Versions/2.3/lib/
  %> cd python2.3/site-packages
  %> otool -L _dbxml.so
    /System/Library/Frameworks/Python.framework/Versions/2.3/Python 
        (compatibility version 2.3.0, current version 2.3.0)
    /Users/jclark/ext/dbxml-2.0.9/install/lib/libdbxml-2.0.dylib 
        (compatibility version 0.0.0, current version 0.0.0)
    /Users/jclark/ext/dbxml-2.0.9/install/lib/libdb_cxx-4.3.dylib 
        (compatibility version 0.0.0, current version 0.0.0)
    /Users/jclark/ext/dbxml-2.0.9/install/lib/libxquery-1.0.dylib 
        (compatibility version 0.0.0, current version 0.0.0)
    /Users/jclark/ext/dbxml-2.0.9/install/lib/libpathan.3.dylib 
        (compatibility version 4.0.0, current version 4.1.0)
    libxerces-c.26.dylib 
        (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib 
        (compatibility version 1.0.0, current version 71.1.1)

For some reason, libxerces-c was linked without path information, unlike all of the other libraries. If anyone knows why this happened, or how to avoid it, please email me or leave a comment. I couldn’t determine how to change this, so I set an evironmental variable to provide the correct search path:

  %> setenv DYLD_LIBRARY_PATH /Users/jclark/ext/dbxml-2.0.9/install/lib
  %> python examples.py test

Success!

As I said earlier, I’m a bit under-experienced with some of the details of building software on *nix systems. If anyone can provide any help with the xerces-c dylib issue, or any improvements to these directions, let me know; I’ll update this post with any additional information I find/receive.

Both comments and pings are currently closed.

3 Responses to “HOWTO: Build DB XML 2.0 with Python Bindings on OS X”

  1. chornbe Says:

    In response to:

    > The README cautions that the python bindings require the pybsddb package, which ships with python 2.3 and later, but which must be compiled against the same version of Berkeley DB as XML DB (the XML DB package builds BDB 4.3.0 from source).
    

    This is the one (and perhaps only) place where (gasp) Windows still outshines the *nux world. Installations are easier, and there are no inter-platform quirks (or darn few, anyway) that cause things like libraries having to be recompiled for xxxx-versions of dependencies.

    I’m actually in favor of certain levels of “commercialization” of some things – one being the software I use when it’s a business scenario. My own tinkering at home is one thing, but in a business environment, I think it’s a little silly to consider recompiling as part of a workstation deployment (for instance) to 1000 desktops.

    Linux rocks, no argument. But it’s got a long way to go (and to come together) to provide for as smooth a user experience as Windows provides (in some areas).

    YYMV

  2. chornbe Says:

    Note: Having said that, I’m getting ready to remove my Windows partition on my main home computer, and am preparing to purchase a new iMac or iBook for my personal use at home as well.

    So, I don’t wanna hear no crap about being a M$ evangelist! :)

  3. ed nixon Says:

    The story continues<br/>

    I’m wondering how you’ve been making out with bdb xml on osx since this was posted. I’ve just spent the last couple of days getting it compiled in order to run under oXygen. The compiling, with Java support, went pretty well after I updated XCode and my Java installation. However, once the code is built, there’s nothing I have found in the docs to tell you about setting up the environment inorder to allow it all to hang together. E.g., I just managed to get the dbxml utility to run after figuring out how to export the dynamic library environment variable you mention above.

    Have you done anything more on this? Have you found any Mac specific resources that bridge some of the gaps that are probably invisible to *nix veterans, developers, etc.?

    Drop me a note if it’s convenient; I’d value anything you can give me.

    thnx. …edN