C++ & embedded Python , Expression Evaluator

[somada141]

Dear Pythoneers,

I have been reading really good answers on this forum so I thought I'll give it a try and see whether someone can help me.

What I want to do:

"I want to embed Python in C++ and use Python's capabilities in order to create a expression evaluator"

In more detail:

"I will have a custom expression entered by the user containing various variables. The values for these variables will be stored in matrices. What I need to do in Python is to use the Py API from C++ to evaluate the result for every set of values for the expression."

What did I do:

After much searching I managed to create a test program for the above problem. Specifically I wrote a simple program that:
->Creates two variables x and y with random values
->Passes those values to two Python variables
->Passes a string containing the expression to Python and evaluates the result
->This result is later converted to a C type and so I have my result

The problem:
The whole thing is slow. After comparing with a Qt implementation of the same problem Qt turns out to be 2times faster than Python.

----Here is the code I used :------

#include <iostream>
#include <cstdlib>
#include <ctime>
#include <string>
using namespace std;

#include <Python.h>
#include <graminit.h>

int main(int argc, char** argv)
{
	const int N = atoi(argv[1]);

    srand(time(0));


    //Create a new namespace dictionary and set the typical built-in values or variables
    Py_Initialize();

    //Creating new namespace dictionary
    PyObject *pDictionary = PyDict_New();
    PyDict_SetItemString( pDictionary, "__builtins__", PyEval_GetBuiltins() );

    PyRun_String( "import math", file_input, pDictionary, pDictionary );
//------------------------------------


    double xValue,yValue;
    string expression="result=math.sqrt(x**2)*math.sin(x)*math.cos(y**2)+math.log(y)*math.sin(x+y)";


    for (int i=0;i<N;i++) {
        xValue=((double) rand()/(double) RAND_MAX);yValue=((double) rand()/(double) RAND_MAX);

        PyDict_SetItemString( pDictionary, "x", PyFloat_FromDouble(xValue) );
        PyDict_SetItemString( pDictionary, "y", PyFloat_FromDouble(yValue) );

        PyRun_String( expression.c_str(), file_input, pDictionary, pDictionary );

        PyObject *pResult = PyDict_GetItemString( pDictionary, "result" );

        double nResult;
        PyArg_Parse( pResult, "d", &nResult );


    }

    Py_DECREF( pDictionary );

	Py_Finalize();

	return 0;
}

First of all let me note that this is the first time I used PyAPI and I might well be doing it wrong.

After the disappointment by the performance I thought of another way:

"What if I somehow convert the entire matrix to a numpy array and then evaluate the expression for all values through a vectorized numpy code...."

This is what I tried to do:

PyRun_SimpleString("from numpy import *");
    PyRun_SimpleString("N=1000000");
    PyRun_SimpleString("result=zeros((N),dtype=float32)");
    PyRun_SimpleString("x=random.random((N))");
    PyRun_SimpleString("y=random.random((N))");
    PyRun_SimpleString("result[0:N]=sqrt(x[0:N]**2)*sin(x[0:N])*cos(y[0:N]**2)+log(y[0:N])*sin(x[0:N]+y[0:N])");

needless to say this code took tens of times less than Python and Qt.

So if there any suggestions on how to convert a C array to numpy and then pass the whole thing to Python (and of course the opposite) I would really appreciate it..

Moreover if anyone can help optimize the above code please do..

Thank you very much for your help

[somada141]

Ok since there is a chance that no-one knows or will actually bother I consider my duty to give a simpler code on how to embed Python in C++ in order to use it as an expression evaluator:

First of all the prerequisites:
1)You need to link to the Python lib:
->That cannot be too clear because the procedure is really depending on the IDE you use. For example I used Code::Blocks but this can be done of course with any IDE (Visual Studio e.t.c)
->You have to go to the project settings or in the case of Codeblocks in the Compiler Settings. In any case after you find the linker settings you have to add the python lib.
->This is placed in the '\libs\ directory under the python directory. For example for me (windows xp) it was in :

'C:\Python25\libs'

and link to the file named 'python25.lib'. Of course the numbers at the end of the filename will change according to your python version(for me I have python 2.5 so the lib is python25.lib)

2)You also need to add the python include dir to the compiler search directories.
->The place where you do that is again dependent on the IDE. You propably have to google this to find out (something like '<name of IDE> add include directories' or sth.)
->The directory you need to include is the '\include' directory in the python folder and for me it was:

'C:\Python25\include'

Well that should do it. Now you can use the code I will give you:

/*
This program merely uses python to calculate the sine of a value passed from C++.

We take a value from C, give it to python, calculate an expression in Python and then
return the result Python found to C++

*/


#include <iostream>
using namespace std;

#include <Python.h>
#include <graminit.h>

int main()
{
    double yValue; //This will hold the variable value

    //This is needed to initialize the interpreter
    Py_Initialize();

    //Create a new namespace dictionary and set the typical built-in values or variables
    PyObject *pDictionary = PyDict_New();
    PyDict_SetItemString( pDictionary, "__builtins__", PyEval_GetBuiltins() );

    //Create a new variable in the namespace dictionary called "y" and set its value to
    //whatever the user wants

    cout<<"Please enter the value (in degrees) for which you want to calculate the sine:\n";
    cin >> yValue;
    double yValue_rad=(yValue/180.0)*3.14;//convert to radians

    //This here creates a new variable in python named 'y' to which the
    //value under 'yValue_rad' is assigned. Of course we have to convert the
    //C type double value to a value python will recognize hence we use
    //the PyFloat_FromDouble function
    PyDict_SetItemString( pDictionary, "y", PyFloat_FromDouble(yValue_rad) );


	//Using our new namespace dictionary to run two python statements.
	//Note how the new variable "y" gets used.

    //we import the math lib to use sin()
    PyRun_String( "import math", file_input, pDictionary, pDictionary );
    //BEWARE: this is the expression evaluation. We make python execute the python code
    //stored in the string we give it. That's what you need to know
    PyRun_String( "result =math.sin(y)", file_input, pDictionary, pDictionary );


	//We pull out the result.
	//pResult is merely a pointer to a string holding the result
	//Note that we we say we want the variable named 'result'
    PyObject *pResult = PyDict_GetItemString( pDictionary, "result" );

	//Here we take the result calculated in Python and convert it to a C type variable.
	double nResult;	//a double that will hold the result
    //This converts the pResult to a double number (that's why we use "d")  and stores it to the referenced variable
	//meaning 'nResult'
	PyArg_Parse( pResult, "d", &nResult );
    cout << "According to Python, sin("<<yValue<<")="<<nResult << "\n"; //We print the result

	//Finally, release everything by decrementing their reference counts.
	Py_DECREF( pResult );
    Py_DECREF( pDictionary );

    //This is needed to finalize the interpreter
	Py_Finalize();

	return 0;
}

I suppose that this is self explained...

Cheers

[Gribouillis]

To convert C arrays to numpy, you need to use numpy's C api. This is explained in numpy's documentation. There are a few functions to create a numpy array. The starting point is here http://numpy.scipy.org/#docs.

[somada141]

Could you possibly direct me to any simple examples or sth? The numpy API documentation is pretty intimidating

[Gribouillis]

I can't help you much, because I never created a numpy array from C, but look at the collection of array creation functions at page 241... of the guide to numpy, and select the most appropriate for your needs :)