Using C++ libraries in Java?

[riahc3]

Hey

Im currently working on a application server written in Java but to use information from a C++ program.

The company seems to agree that rewriting thousands of lines of code would be time consuming and simply making libraries of key functions of the C++ program and then passing them to the Java servlet would be much easier on resources and time (there are only 2 Java programmers)

Searching for information, Ive read about Java Native Interface (or JNI) which allows more or less what we are looking for.

Is this possible and is JNI heading in the right direction?

Thank you

[stevanity]

As far as I know JNI is the way to go. But I think I dont have enough experience to guide you further.

[riahc3]

Is it "difficult" to do?

Most of the C++ libraries (which is currently a program but to be used by Java I imagine they have to be converted into loose libraries) are just math calculations.

[~s.o.s~]

> Is it "difficult" to do?

It depends, but yes, in general I wouldn't shy from using the words "laborious" and "verbose" when it comes to wrapping native code. The process goes something along these lines:

1. Create a new C or C++ project which would act as a "wrapper" for your library
2. This project would contain the "bridge" methods in the sense that they would use the JNI specification to call Java methods from C++ and passing Java invocations to the corresponding C++ methods.
3. This project would depend on your libraries and their respective headers along with the JNI specific stuff. The dynamic library created from this project would be placed on the java library path along with the dependent libraries.
4. In your code, you'll need to load the native library you created before using the functionality provided by your wrapped code.

Apart from this there are a few subtle things which you need to keep in mind like translating from your Java data structures to C++ ones (in the wrapper C++ code), memory management (in case there is a one-to-one mapping between your live instance of your class and a C++ object) and the re-entrant behaviour of your library (can it be safely called from multiple Java threads?).

There are a few helpful libraries out there (both paid and free), which relieve the developer the pain of manual wrapping by providing helper annotations but AFAIK, they are slower than the pure JNI counterpart so definitely not acceptable when it comes to performance intensive applications.

[riahc3]

It depends, but yes, in general I wouldn't shy from using the words "laborious" and "verbose" when it comes to wrapping native code.

Ill explain a bit of the original C++ program :)

Basically it is a drawing program where you set the size of a rectangular space. Afterwards a "paint" like interface is shown, where you draw the shape you want. Once you are finished, the program calculates how many of those shapes can fit next to one another in that rectangular space, also making a drawing of the rectangular space with the shapes already drawn.

If you are satisfied, you press OK the program turns the data into a binary file which then is inserted into a machine which lasers the shapes into a rectangular metal sheet.

More or less and changing a few things around, that is the current C++ program. From what I know, it uses classes.

The process goes something along these lines:

Thanks for explaining :) Hope you can help me on some points as sadly only 2 people know Java in this company which would be me and another guy they hired.

Create a new C or C++ project which would act as a "wrapper" for your library

By this I imagine that you mean that nearly all the calculations in classes, functions, procedures, etc must be turned basically into usable functions in a C++ library right?

This project would contain the "bridge" methods in the sense that they would use the JNI specification to call Java methods from C++ and passing Java invocations to the corresponding C++ methods.

The bridge method is this, right:

extern "C"
 JNIEXPORT void JNICALL Java_ClassName_MethodName
   (JNIEnv *env, jobject obj, jstring javaString)
 {
     //Get the native string from javaString
     const char *nativeString = env->GetStringUTFChars(javaString, 0);
 
     //Do something with the nativeString
 
     //DON'T FORGET THIS LINE!!!
     env->ReleaseStringUTFChars(javaString, nativeString);
 }

The problem is that I see that as C++ read from a Java library. I need Java to read from a C++ library.

This project would depend on your libraries and their respective headers along with the JNI specific stuff. The dynamic library created from this project would be placed on the java library path along with the dependent libraries.

I simply load the C++ library in my Java project and it should load and I should be able to read the functions as I have implanted the JNI standards. Thats what I understood.

In your code, you'll need to load the native library you created before using the functionality provided by your wrapped code.

Same as above.

Apart from this there are a few subtle things which you need to keep in mind like translating from your Java data structures to C++ ones (in the wrapper C++ code), memory management (in case there is a one-to-one mapping between your live instance of your class and a C++ object) and the re-entrant behavior of your library (can it be safely called from multiple Java threads?).

I imagine im going to have to make several new data structures as the C++ are custom structs. Memory management would not be live (I think) and threads they have asked for but I personally know only how to implant them in a mobile environment.

There are a few helpful libraries out there (both paid and free), which relieve the developer the pain of manual wrapping by providing helper annotations but AFAIK, they are slower than the pure JNI counterpart so definitely not acceptable when it comes to performance intensive applications.

This caught my eye as like I said there are C++ programmers in the company but only 2 of us know Java. Could you point out some of those helpful libraries so I can take a look at them and see if they can help out? Our application is online (the C++ does it offline but they want it online using Java; Strange things they are telling us to do....) so performance is not a top issue right now.

[~s.o.s~]

By this I imagine that you mean that nearly all the calculations in classes, functions, procedures, etc must be turned basically into usable functions in a C++ library right?

Basically all classes and functions you create specifically for the purpose of JNI wrapping will go in this project. So you'll have two projects: the first one will be your original C++ library, the second one will be the C++ wrapper project.

The bridge method is this, right:

Yes

The problem is that I see that as C++ read from a Java library. I need Java to read from a C++ library.

Once you are in that function, you are already in C++/native code land. You can freely call any C++ function from your JNI wrapper function and mix-and-match JNI and normal native calls as you please. It is as simple as reading the arguments passed in to the Java method, using them in some way to decide which C++ methods to call. It would be a bit difficult to explain this all here so you might want to read up on some official stuff (i.e. JNI examples by scrouging the internet or get a JNI book).

I simply load the C++ library in my Java project and it should load and I should be able to read the functions as I have implanted the JNI standards. Thats what I understood.

Not really. You load *your* "wrapper library" and the Java runtime will automatically load your C++ library (assuming all of them are avaiable on the library path or the PATH environment variable).

I imagine im going to have to make several new data structures as the C++ are custom structs. Memory management would not be live (I think) and threads they have asked for but I personally know only how to implant them in a mobile environment.

Yes, you are right, you'll have to create wrapper classes for your regular classes (though I'm sure there might be other ways). For e.g. let's say you want to "connect" your Person class in Java to a C++ Person class. You'll in your JNI project create a new C++ class called PersonWrapper which will have a reference to your C++ Person class. You'll have to manage native memory by making sure that the "PersonWrapper" is freed when the corresponding "Person" Java class goes out of scope. The most convinient way to do this would be use finalize methods but finalizers end up slowing your code a bit. We had very highly specced machines so this was never a problem for us but it might be in some cases. PhantomReference class is the new way of clearing native memory tied to a Java instance but requires more work on your part. Regarding threads, we decided to keep it simple by making all methods which modified the global state of the library as synchronized against a static lock. Not the best of approach I agree. A better solution would be to make sure your library doesn't use a lot of global state which will help you do away with the locking/synchronization.

Could you point out some of those helpful libraries so I can take a look at them and see if they can help out?

The free ones I have heard of are: JNA , Swig and HawtJNI; the approach taken is different for all three of them. I haven't used any of them personally since we couldn't afford any sort of penalty in our app on this front (we already had a global lock and finalizers). When making a proof of concept, be sure to check whether the library you use supports all the things required by your project. It would be a pain in the neck if you zeroed on a library only to find down the road that it doesn't support some feature XXX required by your application.

That said, I'd strongly recommend getting a good JNI book or at least spending some time experimenting with the specification.

[riahc3]

Ive followed this tutorial several times

http://java.sun.com/docs/books/jni/html/start.html#769

And at 2.6 it always gives a error about a .dll missing (and it is there) No matter what I modify in the PATH it always asks for another and another....

Is there any other recommended tutorial out there? Most usually copy that one and modify it some....

[~s.o.s~]

And at 2.6 it always gives a error about a .dll missing (and it is there) No matter what I modify in the PATH it always asks for another and another....

I need more details; which part of step 2.6? The part where you try to load your wrapper DLL or the part where you try to create your wrapper DLL?

To create your own wrapper DLL, you'll need to specify the headers (.h files) and libraries (.lib files) for the JVM and your own C++ project.

[riahc3]

I need more details; which part of step 2.6? The part where you try to load your wrapper DLL or the part where you try to create your wrapper DLL?

To create your own wrapper DLL, you'll need to specify the headers (.h files) and libraries (.lib files) for the JVM and your own C++ project.

The first error was that a .dll was missing. That file was "mspdb80.dll". As a fix, we simply added to PATH where it was located and then it gave us other error about stdio.h which we did the same thing or copied it to the same folder as our .C, .H, .class, .java files (we have deleted and redone the tutorial so many times that we have actually forgetten the steps) at last, after working PATH, copying files, including more folders in the -IC:\somefolder, until the linker started erroring out on _CL_randomstufflk.obj files and each time I ran the command it changed the randomstuff part so we are kind of stuff there.

Both of us appriciate very much the help as we are stuck here in a world of pain so to speak.

Thank you very much.

[~s.o.s~]

OK, you should never have to add any random DLL to your path so for the time being forget about manually adding DLL's. How are you building the wrapper project: using IDE or command line? Using Visual Studio? How is your library distributed, as a DLL or LIB?

This is how our project is structured wrt dependencies (Windows):

• Additional libraries included on the path:
  • %JAVA_HOME%\lib\jvm.lib
  • path\to\your\graphics\library\something.lib
• Additional header includes directory:
  • %JAVA_HOME%\include
  • %JAVA_HOME%\include\win32
  • path\your\graphics\project\header\files

[riahc3]

OK, you should never have to add any random DLL to your path so for the time being forget about manually adding DLL's. How are you building the wrapper project: using IDE or command line? Using Visual Studio? How is your library distributed, as a DLL or LIB?

This is how our project is structured wrt dependencies (Windows):

Additional libraries included on the path:
%JAVA_HOME%\lib\jvm.lib
path\to\your\graphics\library\something.lib
Additional header includes directory:
%JAVA_HOME%\include
%JAVA_HOME%\include\win32
path\your\graphics\project\header\files

We are currently trying to do the Hello World program found here: http://java.sun.com/docs/books/jni/html/start.html#769

Nothing more, nothing else.

(In the Hello World program example) we wanted out C++ library to be a .DLL . Since the commands are all done from the command line (in the example), we also ourselves do it from cmd. But like I said, everything comes to a stop at 2.6 with the explaination I gave above.

This is under Windows XP and I installed Visual Studio 2010 Express C++ Edition. As IDE in Java, Im using MyEclipse. The JDK is the latest of version 6, not 7.

[~s.o.s~]

I'm not sure what exactly are you doing there. After reaching home, it took me 30 mins to run the hello world example. Are you sure you are able to properly build the DLL file? The steps in the tutorial might be a bit problematic in case you are using Visual Studio. Did you make sure you add "jvm" headers and "jvm" libraries to the linker when building the project?

You might need to jot down here "what" steps you are taking when using Visual Studio Express Edition to build the DLL down to the minute details.

[riahc3]

I'm not sure what exactly are you doing there. After reaching home, it took me 30 mins to run the hello world example. Are you sure you are able to properly build the DLL file? The steps in the tutorial might be a bit problematic in case you are using Visual Studio. Did you make sure you add "jvm" headers and "jvm" libraries to the linker when building the project?

You might need to jot down here "what" steps you are taking when using Visual Studio Express Edition to build the DLL down to the minute details.

Well the tutorial does say to use cl.exe included in Visual Studio. I dont open Visual Studio at all. I simply use the cl.exe included in the instalation as it says in the tutorial.

As I said, Ill try once again tommorow to do the project again from scratch as see what happens. Ill jot everything down as well.

[~s.o.s~]

Well, surprise surprise, I was able to do it with the command line compiler as well so I guess I'll just show you the steps in brief.

1. Create an eclipse Java project and inside that create a class which has the same contents as that of the tutorial. Just make sure it is in some package like "mypkg" etc.
2. Generate the JNI header file using the "javah" utility as mentioned in the tutorial
3. Create a new folder elsewhere which contains this header file along with the .C file in the tutorial. Make sure that the function definition in the header is the same as the one mentioned in the .C file (which you will have to change since our class is now in a package and package name is part of JNI function signature)
4. Open the shell (CMD) and navigate to the newly created directory.
5. Assuming visual studio is installed at default location; execute C:\Program Files\Microsoft Visual Studio 10.0\VC\bin\vcvars32.bat . This will set the environment variables required for executing cl.exe.
6. Now execute the cl.exe commands specified in the tutorial and make sure you use your own Java path for includes.
7. If everything went out well, you should see a bunch of files in your newly created directory and a HelloWorld.dll
8. Now right click on your class in Eclipse -> Run As -> Run Configurations -> Environment Tab -> Click New -> Add "PATH" as name and "newly created directory location which contains dll" as value -> OK -> Run

That's it! Go ahead and try it out.

[riahc3]

Well, surprise surprise, I was able to do it with the command line compiler as well so I guess I'll just show you the steps in brief.

Thank you very much for the tutorial by you :) Im going to comment on each of your steps...

Create an eclipse Java project and inside that create a class which has the same contents as that of the tutorial. Just make sure it is in some package like "mypkg" etc.

Done. No problems at all

Generate the JNI header file using the "javah" utility as mentioned in the tutorial

Here starts some errors (which have never happened before at this stage). I cleared my PATH for your tutoral so the javah command doesnt work. Im going to add Java´s bin path which should include the javah command. I have my JDK installed at C:\java so I added to PATH C:\java\bin.

After adding that folder I try to run the command and it gives the following error:

error: cannot access HelloWorld
class file for HelloWorld not found
javadoc: error - Class HelloWorld not found.
Error: No classes were specified on the command line. Try -help.

This is normal as in MyEclipse I did not comply/build/run the program. Im going to do that now (as stupid as this may seem, Im try to do everything you posted to make sure I dont do anything wrong so if you dont say comply, I wont)

Tried to run the program and it says (in a window titled "Java Virtual Machine Launcher"

Could not find the main class: mypkg.HelloWorld. Program will exit.

Also console says:

java.lang.UnsatisfiedLinkError: C:\WINDOWS\system32\HelloWorld.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1778)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1703)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1030)
at mypkg.HelloWorld.<clinit>(HelloWorld.java:12)
Exception in thread "main"

The code is this in HelloWorld.java:

package mypkg;

class HelloWorld 
{
    private native void print();
    public static void main(String[] args) 
    {
        new HelloWorld().print();
    }
    static 
    {
        System.loadLibrary("HelloWorld");
    }
}

In my src\mypkg folder (where I am located at in cmd), I have a HelloWorld.java file (just that nothing else).

The error before was because I had a old HelloWorld.dll in my system32 folder. I removed it and now trying to run the program displays this:

java.lang.UnsatisfiedLinkError: no HelloWorld in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1709)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1030)
at mypkg.HelloWorld.<clinit>(HelloWorld.java:12)
Exception in thread "main"

I imagine I have to run the javac command so Im going to do that now.

Done. It generated a .class file. Now Im going to run the "javah -jni HelloWorld" command.

It gives me a error:

error: cannot access HelloWorld
bad class file: .\HelloWorld.class
class file contains wrong class: mypkg.HelloWorld
Please remove or make sure it appears in the correct subdirectory of the classpath.
com.sun.tools.javac.util.Abort
at com.sun.tools.javac.comp.Check.completionError(Check.java:164)
at com.sun.tools.javadoc.DocEnv.loadClass(DocEnv.java:149)
at com.sun.tools.javadoc.RootDocImpl.<init>(RootDocImpl.java:77)
at com.sun.tools.javadoc.JavadocTool.getRootDocImpl(JavadocTool.java:159
)
at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:330)
at com.sun.tools.javadoc.Start.begin(Start.java:128)
at com.sun.tools.javadoc.Main.execute(Main.java:66)
at com.sun.tools.javah.Main.main(Main.java:147)
javadoc: error - fatal error
2 errors

And in the folder of my HelloWorld.java (.../src/mypkg) file, inside there is also a HelloWorld.class file generated with the javac command.

I cant continue the tutorial as I dont have and cant generate the .h file.

Could you help me out?

Again thanks alot :)

[~s.o.s~]

OK, you have to make sure that:

1. javah is in your path (at least temporary; so that you can directly invoke it without being in java\bin)
2. You need to be in a directory just outside the "mypkg" directory which contains your HelloWorld.class file before invoking javah. To do this, search for where the HelloWorld.class is generated (should be autogenerated if build-automatically is on in Eclipse)
3. Open up "CMD" and CD to a directory just one level above the mypkg directory which contains HelloWorld.class.
4. Assuming your class is generated in the following directory strucure "c:\testproject\target\classes\mypkg\HelloWorld.class"; CD to the directory "c:\testproject\target\classes" and then invoke the javah command as "javah -jni mypkg.HelloWorld". There is another way by which you can set the classpath but I'll not mention it to avoid confusion.
5. If you have followed till now, you "should" have a .h file generated in the "classes" directory. If it still doesnt' work; show me your entire CMD session.
6. If it works, continue with the step 3 in my previous post.

[riahc3]

OK, you have to make sure that:
javah is in your path (at least temporary; so that you can directly invoke it without being in java\bin)

It was in path so it worked out of the box

You need to be in a directory just outside the "mypkg" directory which contains your HelloWorld.class file before invoking javah. To do this, search for where the HelloWorld.class is generated (should be autogenerated if build-automatically is on in Eclipse)
Open up "CMD" and CD to a directory just one level above the mypkg directory which contains HelloWorld.class.
Assuming your class is generated in the following directory strucure "c:\testproject\target\classes\mypkg\HelloWorld.class"; CD to the directory "c:\testproject\target\classes" and then invoke the javah command as "javah -jni mypkg.HelloWorld".

Did it and yes it generated my .h file.

If it works, continue with the step 3 in my previous post.

Continuing from step 3....

I arrive at step 5 to run vcvars32.bat,restart cmd and it still doesnt find cl.exe I manually add it to path....

And now I think ive discovered the problem: vcvars32.bat isnt working. Because after adding "C:\Archivos de programa\Microsoft Visual Studio 10.0\VC" where cl.exe is located, it says stdio.h is missing. (which it isnt)

[~s.o.s~]

What do you mean by "restart cmd"? After executing the bat file, you are supposed to use the same session for running cl.exe (assuming you have already added that particular folder to PATH) because changes made by the .bat file execution are transient and carry on only for that given session.

[riahc3]

What do you mean by "restart cmd"? After executing the bat file, you are supposed to use the same session for running cl.exe (assuming you have already added that particular folder to PATH) because changes made by the .bat file execution are transient and carry on only for that given session.

Did not know they only were for that session....

Did it and in the same session ran again the cl line in the tutorial and got this:

Compilador de optimización de C/C++ de 32 bits de Microsoft (R) versión 16.00.30
319.01 para 80x86
(C) Microsoft Corporation. Reservados todos los derechos.

mypkg_HelloWorld.c
Microsoft (R) Incremental Linker Version 10.00.30319.01
Copyright (C) Microsoft Corporation. All rights reserved.

LINK : fatal error LNK1181: no se puede abrir el archivo de entrada 'local\Temp\
_CL_1d74d141lk.obj'

[~s.o.s~]

Sorry, can't help you out on that one; not even sure what that error message means... If you have made any PATH related temporary changes for making this run (e.g. copying DLLs to the current directory etc.), now would be the time to undo those. Do you have multiple VS installations?

Also, paste the command line you are using here.

[riahc3]

Sorry, can't help you out on that one; not even sure what that error message means... If you have made any PATH related temporary changes for making this run (e.g. copying DLLs to the current directory etc.), now would be the time to undo those. Do you have multiple VS installations?

Also, paste the command line you are using here.

My PATH currently contains:

%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\Archivos de programa\ATI Technologies\Fire GL 3D Studio Max;C:\java\bin

Nothing strange.

My command line is:

cl -Ic:\java\include -Ic:\java\include\win32 -MD -LD mypkg_HelloWorld.c -Femypkg_HelloWorld.dll

when I am located in the folder containing both the mypkg_HelloWorld.c and mypkg_HelloWorld.h

The c:\java\include and Ic:\java\include\win32 exist. Im using Visual Studio 2010 and none other.

[riahc3]

Visual Studio 2010 C++ Express.

[~s.o.s~]

What is the location of the directory from which you are invoking "cl.exe"? Also, can you try putting the -I paths in quotes? If that doesn't work, can you try:

cl.exe -I "c:\java\include" -I "c:\java\include\win32" -MD -LD mypkg_HelloWorld.c -Femypkg_HelloWorld.dll

[riahc3]

What is the location of the directory from which you are invoking "cl.exe"? Also, can you try putting the -I paths in quotes? If that doesn't work, can you try:

I am invoking it from "C:\tutorial" where I have mypkg_HelloWorld.c and mypkg_HelloWorld.h

I tried what you put:

cl.exe -I "c:\java\include" -I "c:\java\include\win32" -MD -LD mypkg_HelloWorld.c -Femypkg_HelloWorld.dllcl.exe -I "c:\java\include" -I "c:\java\include\win32" -MD -LD mypkg_HelloWorld.c -Femypkg_HelloWorld.dll

and still nothing. Same error.

[riahc3]

Sorry....

cl.exe -I "c:\java\include" -I "c:\java\include\win32" -MD -LD mypkg_HelloWorld.c -Femypkg_HelloWorld.dll

[~s.o.s~]

What does this exactly mean in English?

LINK : fatal error LNK1181: no se puede abrir el archivo de entrada 'local\Temp\
_CL_1d74d141lk.obj'

[riahc3]

What does this exactly mean in English?

LINK : fatal error LNK1181: no se puede abrir el archivo de entrada 'local\Temp\
_CL_1d74d141lk.obj'

Sorry thought you just didnt know the error code.

no se puede abrir el archivo de entrada 'local\Temp\
_CL_1d74d141lk.obj'

The file 'local\Temp\_CL_1d74d141lk.obj' cannot be opened.

[~s.o.s~]

This is definitely something specific to your machine and related to spaces in path or newlines. But don't think I would be able to pin-point the solution sitting here.

Let's try another approach; can you setup the same thing in Visual studio?

[riahc3]

This is definitely something specific to your machine and related to spaces in path or newlines. But don't think I would be able to pin-point the solution sitting here.

Let's try another approach; can you setup the same thing in Visual studio?

Could you be more especific when you say "setup the same thing in Visual studio"?

Currently Im not at work anymore (its currently 15:31) and I enter tommorow at 09:00, so Ill read and do it tommorow :)

[~s.o.s~]

> Could you be more especific when you say "setup the same thing in Visual studio"?

Simply set up a new DLL C project in Visual studio. Create new blank project. Right click project -> General -> Configuration Type set to Dynamic Library DLL

Right click Header folder -> add item -> add your header

Right click Source folder -> add item -> add your .c file

Right click project -> c/c++ -> Additional include directories -> add paths to java\include and java\include\win32

Right click project -> rebuild -> the output window will show you the location of the newly created dll

For more detailed instructions, you should probably speak with the C++ guys from your team since they should definitely be able to help you out with this one.