Quantcast
Channel: Windows Inspired
Viewing all articles
Browse latest Browse all 7

How to Step Through an Executable Program’s Startup Code in Visual Studio

$
0
0

What saves a man is to take a step. Then another step.

This post shows you how to step through the startup and initialization code of a Windows program using Visual Studio.

Introduction

From a programmer’s perspective, a Windows application written in C or C++ begins execution at main() or WinMain(). But there’s a lot of initialization that occurs before either of these functions is called. Among other things, the runtime library (CRT) is initialized and constructors are called for C++ global objects.

It is sometimes useful and relatively easy to step through this initialization code and I will discuss several techniques for doing it.

How you go about this depends on what you want to look at, the availability of source code, and whether or not there is debugging information. I will describe techniques you can use with programs you developed yourself (and have source code for), as well as executables you did not build.

You probably don’t need to be shown how to trace such things as constructor calls for classes that you developed yourself and for which you have source code. All you have to do is set breakpoints in the constructors and restart the program. This is trivial so I won’t bother to show how to do it.

It’s only slightly more difficult to step through the CRT (C or C++ Runtime Library) startup code. Why would you want to do this? Possibly to understand how a program’s command line is parsed to generate argv[], or learn the mechanism for calling global constructors or scheduling destructor calls for the same objects. All of this occurs before main is called.

Visual Studio provides source code for the CRT, located by default at C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\crt\src. This is a valuable resource. But just reading the sources is no substitute for actually stepping through the code at runtime. You can even modify and compile these sources to add breakpoints and debugging messages.

I assume you have Visual Studio installed. I am using Visual Studio 2010 but everything should work (with minor differences) with Visual Studio 2008 or Visual Studio 2012, and probably even earlier versions.

As with class constructors, if the code you want to trace is in the CRT and the program was built with debugging information you can open one of the CRT source files and set a source-level breakpoint— provided you know what you want to trace and in which source file it is located. Because you are not likely to be as familiar with this code as with your own, and because it employs some advanced techniques (like calling initialization functions through arrays of function pointers that are populated based on specially-named code sections), you will likely want to step through some of this code starting at the program entry point.

Again, I will assume you know how to set source-level breakpoints and instead concentrate on what to do if you have just an executable image with no debugging information. The program might not even link with the CRT or might have a custom entry point, something other than MainCRTStartup.

Launching the Visual Studio Debugger

The following are the general steps you must follow to get things set up:

  1. Create a Visual Studio solution for the program and open it (this happens automatically for most of the methods shown below)
  2. Find the memory address of some piece of code that runs before the code you want to step through
  3. Set a breakpoint at this address
  4. Start the program so the breakpoint can be hit and begin stepping through code

We need to open a Visual Studio solution associated with the program’s executable image. This is not the same as attaching to a process, though attaching to a process is one way it can be achieved. We eventually want the program to be opened in Visual Studio but not yet running so that we can step through its initialization. Logically, since the program will not initially be running, there is no process to attach to.

Even though we ultimately want to start running the program from the beginning, there is one piece of information we can’t get without the program running or at least stopped at a breakpoint– its base load address. So each of the following options will leave the executable either running or stopped at a breakpoint. Once we have the load address we will restart the program.

The task is easiest if you already have a Visual Studio solution for the program. Just open it and press F11 (step into) to start execution and immediately break.

If you don’t already have a Visual Studio solution, there are several ways you can create one automatically. Use whichever of the following methods you feel comfortable with:

1) Create a debug breakpoint by adding the following statement to the program (this is for those of you who are building outside of Visual Studio, otherwise it’s easier to just use the VS solution):

__debugbreak();

Compile and run the program.

The breakpoint will cause the familiar Program has stopped working dialog:

 

Since you have Visual Studio installed there will be a Debug the program button.
 
Press it.

   
You will be presented with a dialog asking you to select a debugger.

Choose New Instance of Microsoft Visual Studio and click Yes.

 
When you see a Program has triggered a breakpoint message press Break.
 
Visual Studio will open, with the program stopped at the breakpoint.

 
2) If the program is already running, open Visual Studio and from the Debug menu select Attach to Process. Select your program’s process from the dialog that pops up. You can leave the program running or break into it by clicking Debug->Break All (Ctrl+Alt+Break).

3) If you do not have a Visual Studio solution and the program executes too quickly to allow attaching to the process, you can use the following technique to cause Visual Studio to automatically attach to the program when it is started:

  • Open the registry editor, regedit.exe.
  • Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
  • Right-click Image File Execution Options and add a new key with the executable’s name (no path). Example: “TestApp.exe” (without the quotes)
  • Right-click the new key and add a new string value named “debuger”, and set the value of the string to “vsjitdebugger”
  • After this, any time an executable named TestApp.exe is started (by any mechanism), the Visual Studio debugger will automatically open even if the program terminates quickly.
  • This method does not automatically generate a breakpoint, so the program will be still be running if it has not yet terminated.
  • If the program has terminated, press F11 to restart it and break immediately.
  • If you are working with several executable programs you can rename the key you just created instead of polluting the registry with multiple keys.

Whatever technique you use, at this point Visual Studio will be open with the program either running or stopped at a breakpoint.

Finding the Program Entry Point

You usually can’t just press F11 to step into a program’s entry point from a completely stopped state. If Visual Studio can find main or WinMain, it steps over the initialization and stops at the first statement of main or WinMain (unless a breakpoint is encountered first). This is one of the rare situations where having debug information/symbols available actually hinders you. If debug information is not available, pressing F11 does indeed step into the first assembly language instruction and stops at the program entry point. In theory you can avoid having to find the entry point by building your software without debug information and just pressing F11 to start it. I doubt the little bit of effort you’ll save is worth the pain of not having symbols.

footnote: I have to admit I am not certain there isn’t a way to force Visual Studio to always start in Disassembly mode. If anyone knows how please post a comment.

So let’s assume we need to find the program’s load address and entry point.

Every Windows executable has an entry point which is specified in the header of the executable file. I’ll show you first how to find the entry point using the DumpBin utility that comes with Visual Studio, but there is an easier way which I’ll also show. Make sure you use the version of DumpBin from the 64-bit compiler or you won’t see the correct information for a 64-bit executable (the 64-bit version works correctly for both 32-bit and 64-bit images).

From a command prompt run vcvars64.bat (by default in c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\vcvars64.bat) then run the following command:

DumpBin /headers TestApp.exe

Look for “entry point” and “image base” which will look something like this for a 32-bit executable (these 2 lines are not necessarily consecutive in the output):

1690 entry point (00401690) _mainCRTStartup
400000 image base (00400000 to 00409FFF)

Or this for a 64-bit executable:

22EB8 entry point (0000000100022EB8)
100000000 image base (0000000100000000 to 0000000100660FFF)

This information is not used directly. The actual address depends on the load address of the program (which can change on every run). To get the actual address you first need to find the base load address then calculate the entry point relative to it. Don’t worry, it’s easier than it sounds!

Note: there are other ways to get the image base address if you build the program yourself (for example, from the linker configuration or the .map file generated by the build) but I’m not going to discuss them.

To get the actual load address, open the Modules window from Debug->Windows->Modules menu and locate the executable under the Name column. Then get the load address from the Address column. The program must be running or stopped at a breakpoint.

From the DumpBin information you can calculate the offset of the entry point relative to the base address (I’ll use the numbers from the above 64-bit executable):

Relative entry point = entry point – base address = 0x0000000100022eb8 – 0x0000000100000000 = 0x0000000000022eb8.

This means the real entry point in memory is 0x00022eb8 bytes after the real load address. If the Modules window shows the program is loaded at 0x00000000ff290000, then the entry point is 0x00000000ff290000 + 0x0000000000022eb8 = 0x00000000ff2b2eb8.

Setting the Breakpoint

Once you have the entry point you can enter it into the Address box at the top of the Disassembly window. This will display the entry point and you can click the left margin next to the instruction to set a breakpoint.

After setting the breakpoint you can stop the current execution, then restart the program by pressing F5. Note: if the executable does not have debugging information, you may see a message asking if you want to proceed. Just click Yes.

Houston, We Have a Problem…

We come now to a big problem with this technique called Address Space Layout Randomization (ASLR).

Every time you restart the program (even within a single debug session) the load address can change. Which means the breakpoint you just set will no longer point to the desired instruction. There are ways to disable ASLR (at build time for a specific program or for your system as a whole) but I don’t recommend them.

Instead, you can load another instance of your program and keep it loaded (either by letting it run forever or by stopping it at a breakpoint). All subsequent instances, including the one you are debugging, will load at the same address. Be aware that this has been the case in my experience, but I have not seen it documented and can’t claim it is always true. Now when you stop and re-start your program, the breakpoint address will still be valid.

If the application you are investigating is a single-instance program, then the technique of loading another instance to keep the load address constant can usually still be used. Windows always loads subsequent instances of the program and the program itself detects that a prior instance is running and exits. This is typically done inside WinMain, which runs after the part of the program you want to step through.

Finding the Program Entry Point and Setting the Breakpoint, Take 2

There is another method for getting to the entry point that I personally find easier, and which also avoids the ASLR problem. You set a breakpoint in kernel32.dll at the point where it calls the entry point of a new process. From that breakpoint you can single step into your app’s entry point.

There’s a few issues with this method that you need to watch for. I will point these out as we go along.

To begin, press F11 to start the program (it will break immediately) and examine the call stack.

Look for the entry in kernel32.dll just below the earliest stack frame for your executable. This is where the Windows calls the entry point to the program. If you double click it, you will be brought to where the entry point is called from kernel32.dll. The small green arrow represents where your program will return to kernel32.dll.

You can set a breakpoint on the call (the “call rdx” instruction in the image), stop the current execution, then restart by pressing F5. When the break point is hit, you can step into the entry point, which here is at memory address 0x00000000ff2b2eb8 (the yellow arrow).

The first thing to be aware of is that for some reason when you stop the program (Debug->Stop Debugging, or Shift+F5) the breakpoint in kernel32 always becomes disabled. I’m not sure why (though I have some ideas). You must always re-enable the breakpoint right before restarting. Do this by clicking the breakpoints tab, then click the checkbox next to the breakpoint. The second issue is that you can’t start by pressing F11. You must press F5 or the breakpoint becomes disabled and is not hit.

One advantage of this approach is that the address inside kernel32.dll of the call to the entry point seems to be the same for all processes, so you don’t have to run a second instance of the program you are testing. Presumably this is because all processes share a single instance of system DLLs such as kernel32.dll. I have not rigorously checked this fact. It is theoretically possible for Windows to map a single instance of a DLL in physical memory to different virtual addresses in different processes, but I believe this is not done (probably for CPU cache-efficiency reasons) and the limited checking I did has shown it to always be loaded at the same address.

Final Note

This was not an exhaustive treatise on the subject, but I hope it gets you going in the right direction.


Viewing all articles
Browse latest Browse all 7

Latest Images

Trending Articles





Latest Images