Using x86 Assembly Language with Microsoft Visual C++

[meabed]

Creating a C++/Assembly Project
The project we'll develop in this tutorial will consist of a main() function written in C++. It will call an assembly function named clear(). Since Visual C++ does not recognize assembly code, VC++ will have to be told what program to call to compile the assembly code. In this section, we'll cover the basic steps of creating a VC++ project, adding assembly code to it, specifying the Custom Build instructions, and building the project.
Step 1- Create Project
Create a standard Visual C++ project. If you are unsure of how to do this, check the Visual C++ Introduction page. Add a cpp source file with a main() function. For this tutorial, we will create a main() function that will call an assembler function named clear(). clear() is of type void and requires no parameters and will be located in a separate file called clear.asm. Since it is in a separate file, clear() will need to be declared at the beginning of the file. Our main() function will look like this:

// Declaration of assembly function 
// extern "C" instructs compiler to use C calling conventions 
extern "C" void clear(); 

int main() { 
clear(); 
return 1; 
}

The phrase "extern "C"" in the declaration tells the compiler to use the C calling convention. This effects things like how parameters are passed to the routine and whether the calling function or the called function has responsibility for saving and restoring the registers. Most importantly for us, it prevents name-mangling. When C++ code is compiled, the names of functions are changed drastically to include extra information. This is commonly called name-mangling. The language C also changes the name, but it merely prepends an underscore to the front of the name. The "extern "C"" tells the compiler to do this simpler, C-style name change instead of the more complex, standard C++ name-mangling.
Leaving out the "extern "C"" will result in an error like:

asm_tut.obj : error LNK2001: unresolved external symbol "void __cdecl clear(void)" (?clear@@YAXXZ)
Step 2- Add Assembly Code
Add the file that contains your assembly source code to the project. If this hasn't been created yet, you can do this by selecting FileView in the Project Window, right-clicking on the project's name and selecting "Add files to project..." When the dialog box appears, type in the name you want the assembly code file to be saved as (in our case, clear.asm). VC++ will warn you that the file does not exist and ask if you want to create a reference to it in the project anyway. Select Yes. Expand the tree listing in the project window until you see the name of the assembly file (clear.asm). Double-click the file name. VC++ will ask if you want to create a new file with that name. Select Yes. A new file will be created and opened in the editor.
Enter you assembly code. For this tutorial, we will clear the EAX and EBX registers. To do this, we'll use this code:

.586              ;Target processor.  Use instructions for Pentium class machines
.MODEL FLAT, C    ;Use the flat memory model. Use C calling conventions
.STACK            ;Define a stack segment of 1KB (Not required for this example)
.DATA             ;Create a near data segment.  Local variables are declared after
                  ;this directive (Not required for this example)
.CODE             ;Indicates the start of a code segment.


clear PROC 

xor eax, eax 
xor ebx, ebx 

ret 

clear ENDP 
END

Step 3- Set Custom Build Commands
We now provide the commands that VC++ will use to compile the assembly code. Visual C++ does not compile assembly source code. It must call an external program named ml.exe to perform the compilation. The commandline must be added to Custom Build options of the Project Settings. To do this, right-click on the assembly filename (clear.asm) in the Project window. Select "Settings..." from the pop-up menu. Select the "Custom Build" tab. In the "Commands" text box, type

y:\vstudio\masm611\bin\ml /c /Cx /coff $(inputpath)
if you are working in the Olsson laboratories. The command for the stacks is

g:\apps\win32\masm611\bin\ml /c /Cx /coff $(inputpath)
THE SWITCHES IN THIS COMMAND ARE CASE SENSITIVE. /c and /C are very different and will create errors. The string $(inputpath) is a macro which expands to the path and name of the assembly code file.

Under "Outputs", type:

$(InputName).obj
This tells VC++ that the results of the Custom Build will be an obj file with the same name as the assembly file (e.g., our sample project will generate a file named clear.obj which will be linked into the final program). The Dialog box should look like this:
[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]
Select "OK" to close the dialog box.

Step 4- Compile and Link
The project may now be compiled, linked, and run like any other VC++ project. Press F7 to Build the project and F5 to execute the program.
Debugging a C++/Assembly Project
Debugging a program with assembly is a little more involved than debugging a pure C++ program. In particular, Step-Into (F11) will not descend into an assembly module. We must use the "Disassembly Window" to get around this limitation. To debug the assembly code, we will have to stop the program with a breakpoint, open the disassembly window to view the assembly code, open the register or the memory windows, and use F11 to step through the code in the disassembly window.
Step 1- Stop the Program
Place a breakpoint on the line of your code that calls the assembly routine. Run the main C++ program by pressing F5 (Go). It will execute normally and stop when it hits the breakpoint.
Step 2- Open the Disassembly Window
Open the Disassembly Window by selecting View->Debug Windows->Disassembly from the menu. A new window will appear that should look like this:

[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]
The Disassembly Window shows the underlying assembly instructions for C++ instructions. The actual C++ code we wrote is listed in black. The disassembled code is listed in grey after the C++ instruction. These are the actual assembly instructions which will be executed as the program is run. The yellow arrow, indicating the next instruction to be executed, is present in this window. The arrow, however, is not pointing to the C++ instruction clear(), but rather to the assembly instruction

00401038 call @ILT+0(_clear) (00401005)
The Disassembly Window allows stepping through code line-by-line just as the normal debugger window does; however, it traces the execution of assembly instructions instead of the higher C++ instructions.

Step 3- Display Registers and Memory
You may want to open the Register and Memory Windows at this point. These windows provide a snapshot of what's in the CPU's registers and system's memory between execution of program instructions. Open the Register Window by selecting View->Debug Windows->Registers from the menu. It should appear like this:

[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]
You can see the register values for EAX, EBX, ECX, EDX, ESI, EDI, ESP, and EBP, as well as some other registers and status flags present in the processor (these registers are from a Pentium Pro; registers in other processors may be different although the 8 listed will be present).

To examine memory, open the Memory Window in the same manner. It appears like this:

[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]

It provides a memory dump with the memory address on the left, the hexadecimal values of the memory contents on the right, and the ASCII representation of the hex values on the right. A particular memory location can be displayed by typing the address in the text box at the top of the window. The window shown above displays the beginning of our program. The first instruction is in memory location 0x00401020 and is 0x55. This is the hexadecimal encoding of 'push ebp'. The next six numbers on the line show subsequent memory locations. A total of seven memory locations is shown on each line in this window. This is based on the size of the window. The final column is the ASCII characters for the memory locations. This will usually be garbage unless you are viewing a memory region that has text stored in it.

Step 4- Step through the Code
Single-step through the code using F10 (Step Over) and F11 (Step Into) as if it were a normal program. The Disassembly Window will trace execution of assembly instructions with the yellow arrow pointing to the next instruction to be executed.
Pressing F11 once executes the call instruction. We're now at a jmp instruction that will jump us to beginning of the clear() function.

Pressing F11 again takes us to the first instruction of the actual clear() function. The Disassembly Window now looks like this:
[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]

Notice that the yellow arrow is pointing to the first of our two xor calls. The Registers window at this point is unchanged. Pressing the F11 key again executes the first xor statement, clearing the EAX register. The Registers window is now:

[IMG]http://www.cs.virginia.edu/~csadmin/pc/pclabs/vc_faq/img/asm_dialog.gif[/IMG]

The EAX is now 0x0. Pressing F11 again clears the EBX register. Pressing F11 again returns from the clear() function and places us below our C++ command "return 1;". Since we've finished debugging the crucial part of our code, we can press F5 to Go and quickly finish the program.

[Chainsaw]

That's an excellent primer! Thanks!

For very simple routines, you can also inline assembler, like this:

// return a+b
int AddTwoIntegers( int a, int b )
{
    int result;

    __asm
    {
        mov eax, a  // inlined assembler can deal with simple local variable references
        add eax, b
        mov result, eax
    }
    return result;
}