Using a c++ library in python

[Andy_Ballard]

Hello,

I would like to import and call a c++ library from python.
The library is the freely available (c++) LAMMPS Molecular dynamics code, and on the website, the following advice is given:

"[one can]...build LAMMPS as a library. Once this is done, you can interface with LAMMPS either via C++, C, or Fortran (or any other language that supports a vanilla C-like interface, e.g. a scripting language). "

I have succeeded in building the library, but would like some advice on how to proceed from here.
Many thanks
Andy

[Gribouillis]

You need to write interface code between your C++ library and python. A good tool for this is the swig interface generator which is well documented.

If you know C++, would you be able to write a small program which uses your library and does something useful ? In that case make a list of the library functions that you are using in your program and start writing a swig interface file with these functions, then write the same program in python. This would give you a starting point, because such C++ libraries often have dozens of classes and functions.

[Andy_Ballard]

You need to write interface code between your C++ library and python. A good tool for this is the swig interface generator which is well documented.

If you know C++, would you be able to write a small program which uses your library and does something useful ? In that case make a list of the library functions that you are using in your program and start writing a swig interface file with these functions, then write the same program in python. This would give you a starting point, because such C++ libraries often have dozens of classes and functions.

Thanks,
This gives me a good starting place, and will try what you suggest.

[Stefano Mtangoo]

And don't forget, you can use Foreign library interface; The so called ctypes
It comes with Python (>=2.5) and It is in Docs

[Andy_Ballard]

I have made some progress in the recommended direction, but am currently stuck...

I now have a simple C program which uses some of the functions of the library which I mentioned earlier. These are included in the C program with an #include "library.h"
I can successfully compile this program by giving it the required libraries:
g++ lmp_driver.c -I/home/andy/lammps/src -L/home/andy/lammps/src -llmp_g++

To create the python module out of this C file, I followed the manual, and wrote a header file, lmp_driver.h

/* File: lmp_driver.h */
int main(int narg, char **arg);

and a swig interface file, lmp_driver.i

/* File: lmp_driver.i */
%module LmpDriver

%{
#define SWIG_FILE_WITH_INIT
#include "lmp_driver.h"
%}

int main(int narg, char **arg);

I then make it with the following includes and links

CC=g++
CFLAGS= -O2 -fpic -c
CINC = -I/home/andy/lammps/src
CLINK = -L/home/andy/lammps/src
CLIB = -llmp_g++

SWIG = swig -python
PYINC = -I/usr/include/python2.6

all: lmp_driver.o lmp_driver_wrap.o
	echo "shared library"
	$(CC) -shared lmp_driver.o lmp_driver_wrap.o \
	-o _lmp_driver.so

lmp_driver_wrap.o: lmp_driver_wrap.c
	echo "compiling(?) the wrapper file"
	$(CC) $(CFLAGS) lmp_driver_wrap.c \
	$(CINC) $(CLINK) $(CLIB) $(PYINC)

lmp_driver_wrap.c:
	echo "using swig with the interface file"
	$(SWIG) lmp_driver.i

lmp_driver.o: lmp_driver.c
	echo "compiling the lammps .c file"
	$(CC) $(CFLAGS) lmp_driver.c \
	$(CINC) $(CLINK) $(CLIB)

And this makes without errors, but then when I try to import into python ("import _lmp_driver"), I am left with the error

ImportError: ./_lmp_driver.so: undefined symbol: lammps_close

The error mentions a library function which is included in the C program through the use of a header file.

I think that somehow it has not been linked correctly in the make process. Any ideas how this can be corrected?

[Gribouillis]

Don't you think $(CLINK) and $(CLIB) should be given to the build of _lmp_driver.so instead of lmp_driver.o ?

[Andy_Ballard]

Don't you think $(CLINK) and $(CLIB) should be given to the build of _lmp_driver.so instead of lmp_driver.o ?

lmp_driver.o doesn't compile without $(CLINK) and $(CLIB).
Giving them to the build of _lmp_driver.so *as well* builds successfully. Python is now giving
ImportError: dynamic module does not define init function (init_lmp_driver)

I don't quite follow this either, since I thought that the line
#define SWIG_FILE_WITH_INIT
in the lmp_driver.i file was defining the init

[Gribouillis]

It's a classical error, look at section 31.2.5 in your swig doc. You should probably write %module lmp_driver

[Andy_Ballard]

That final error was much easier to solve after reading the swig manual again

I had mismatched the module name at the top of the .i file with the .so file name.

Now it is working, with the required changes being that the file lmp_driver.i needed a name change:

/* File: lmp_driver.i */
%module lmp_driver

%{
#define SWIG_FILE_WITH_INIT
#include "lmp_driver.h"
%}

int main(int narg, char **arg);

Thanks for the assistance,
Andy

[Gribouillis]

Thanks. If you manage wrapping some parts of the library with swig, don't hesitate to post your swig file here. It's always interesting to see a successful binding of a library to python.

[Andy_Ballard]

I managed to wrap this library, and I almost feel embarrassed that the interface file which took me so long to figure out is so short. I did have to spend a while with mpi4py though, since I will need MPI for further coding.

//File library.i

%module library
%{
#define SWIG_FILE_WITH_INIT
#include "library.h"
%}

%include mpi4py/mpi4py.i
%mpi4py_typemap(Comm, MPI_Comm);

%include "carrays.i"
%array_functions(int, iArr);
%array_functions(double, dArr);
		
//%pointer_functions(double, dPtr)

//Allowing us to pass c-style pointers thru python
%include "cpointer.i"
%pointer_functions(int, intp);
%pointer_functions(double, dPtr);

//Allowing us to malloc that pointer
%include "cmalloc.i"
%calloc(double)

//a typemap so that argc argv can be passed
%typemap(in) (int narg, char **arg) {
  /* Check if is a list */
  if (PyList_Check($input)) {
	int i;
	$1 = PyList_Size($input);
	$2 = (char **) malloc(($1+1)*sizeof(char *));
	for (i = 0; i < $1; i++) {
	  PyObject *o = PyList_GetItem($input,i);
	  if (PyString_Check(o))
	$2[i] = PyString_AsString(PyList_GetItem($input,i));
	  else {
	PyErr_SetString(PyExc_TypeError,"list must contain strings");
	free($2);
	return NULL;
	  }
	}
	$2[i] = 0;
  } else {
	PyErr_SetString(PyExc_TypeError,"not a list");
	return NULL;
  }
}

%typemap(freearg) (int argc, char **argv) {
  free((char *) $2);
}

%include "library.h"

which was made, including my library called lmp_openmp, as follows:

.PHONY: default
default: clean build test

PYTHON = python

#This will hopefully discover your python automatically
PYTHON_INC = ${shell ${PYTHON} -c 'from distutils import sysconfig; print( sysconfig.get_python_inc() )'}

#Discover the path of your mpi4py
MPI4PY_INC = ${shell ${PYTHON} -c 'import mpi4py; print( mpi4py.get_include() )'}

#Find your dll extension
SO = ${shell ${PYTHON} -c 'import imp; print (imp.get_suffixes()[0][0])'}
SWIG = swig
SWIG_PY = ${SWIG} -python -c++

#Compiler, compiler flags
MPICC = mpic++
CFLAGS = -O2 -fPIC -c 

#lammps library location and name
CINC = -I /home/andy/lammps/src
CLINK = -L /home/andy/lammps/src
CLIB = -llmp_openmpi

.PHONY: build
build: _library.so
_library${SO}: library_wrap.o library.o
	$(MPICC) -shared library.o library_wrap.o -o _library.${SO} \
	$(CINC) $(CLINK) $(CLIB)

library.o: library.cpp library.h
	$(MPICC) $(CFLAGS) library.cpp $(CLINK) $(CINC) $(CLIB)

library_wrap.o: library_wrap.cxx
	$(MPICC) $(CFLAGS) library_wrap.cxx $(CINC) $(CLINK) $(CLIB) \
	 -I${PYTHON_INC} -I${MPI4PY_INC}

library_wrap.cxx: library.i
	$(SWIG_PY) -I${MPI4PY_INC} -o $@ $<

.PHONY: clean
clean:
	${RM} $(CODE).py* _library.${SO} *.cxx *.o

.PHONY: test
#If it is working, this will give python error:
	#TypeError: lammps_open() takes exactly 4 arguments (0 given)	
test: 
	${PYTHON} -c "import library; library.lammps_open()"

Thanks a lot for the assistance, if anyone knows good resources for beginners (apart from the reference manual) for understanding mpi4py I would be receptive.
Andy