Click here to Skip to main content
15,867,308 members
Articles / Containers / Virtual Machine
Tip/Trick

Kernel-Mode Debugging in a VM using Visual Studio 2012

Rate me:
Please Sign up or sign in to vote.
4.80/5 (7 votes)
31 Aug 2013CPOL4 min read 58.8K   23   14
This tip is about how to setup Kernel-Mode Debugging in a VM using Visual Studio 2012.

Introduction

This tip is about how to setup Kernel-Mode debugging in a VM using Visual Studio 2012. This tip will be helpful for people new to driver development that wish to create, build, deploy, and debug a simple "HelloKernel" project.

Background

The setup process can be divided into three parts:

  • Setting up the Virtual Machine
  • Setting up Visual Studio 2012
  • Setting up your HelloKernel project

Each part will be explained step by step.

How to Setup the Virtual Machine

The Virtual Machine setup was tested on VMWare 9.0 running Windows 7 x64 OS.

  • Create a new VM, use default settings (tested on VmWare, Win7 x64)
  • Add a serial port to the VM, follow these steps:
    • Right click VM -> Settings -> Add -> Serial Port -> Output to named pipe
    • Named pipe = \\.\pipe\com_2
    • First combo box = The end is the server
    • Second combo box = The other end is an application
    • Connect at power on = Checked
    • Finish
    • Yield CPU on poll = Checked
  • In the VM, in an elevated command prompt window, enter the following commands:
    • bcdedit /debug on
    • bcdedit /dbgsettings serial debugport:2 baudrate:115200
  • In Windows, search for "Manage advanced sharing settings":
    • Set "Turn on file and printer sharing"
    • Set "Turn off password protected sharing"
  • In Windows, search for "Computer Management":
    • Local Users and Groups -> Users:
    • Administrator -> Right Click -> Properties -> Uncheck Account is disabled
    • Administrator -> Right Click -> Set Password -> Set your password
  • Install WDK Test Target, the setup can be found on the host computer in the following path:
    "C:\Program Files (x86)\Windows Kits\8.1\Remote"
  • Reboot the VM

How to Setup Visual Studio

  • Install Visual Studio 2012 (or above) and WDK 8 (or above)
  • Run Visual Studio as administrator
  • On the main toolbar, press Driver-> Test -> Configure Computers -> Add New Computer:
    • Computer name = VM name
    • Select Provision computer and choose debugger settings
    • Next
    • Connection Type = Serial
    • Pipe = Checked
    • Reconnect = Checked
    • Port = \\.\pipe\com_2
    • Target Port = com2
    • Next -> Finish (Don’t worry if Status is Undetermined)
  • Reboot the VM
  • Create a snapshot in the VM called "Configured"
  • You may be prompted for Username\Password, enter Administrator\YourPassword
  • Important! Currently Visual Studio 2012 has a weird defect that causes the steps above to fail if your Windows username has spaces in it. The active username must have no spaces !

How to Setup Your HelloKernel Project

  • Use this tutorial to create your HelloKernel project:
  • In order to debug your project, do the following:
    • OPTION I
      • Press F6 (Important! A newly configured VM requires one successful build before debugging)
      • Press F5
      • When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • In the "Debugger Immediate Window" write the following commands:
      • bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • g (Tells the debugger to continue)
    • OPTION II
      • Open two instances of VS, one will be the Deployer and the second the Debugger
      • [In Debugger] On the main toolbar press Debug -> Attach to Process
      • [In Debugger] Select "Windows Kernel Mode Debugger" and attach to the VM
      • [In Debugger] When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • [In Debugger] In the "Debugger Immediate Window", write the following commands:
      • [In Debugger] bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • [In Debugger] g (Tells the debugger to continue)
      • [In Deployer] Press F6
    • OPTION III
      • Setup VS as a Debugger, as done in OPTION II
      • Copy the package folder located at "HelloWorld\Win7Debug\HelloWorld Package" into the VM
      • In the VM, create a new file named install.bat with the following commands:
      • sc stop HelloWorld
      • "C:\DriverTest\devcon.exe" install "C:\HelloWorld Package\HelloWorld.inf" Root\HelloWorld
      • sc start HelloWorld
      • In the VM, run install.bat as administrator
      • If prompted, press "Install this driver software anyway"
      • In order to reinstall a newer version of the driver, just update the package folder and rerun install.bat
  • Your code should break on DriverEntry, and you can start debugging

Tips

  • Tip I - Be sure your solution configuration is valid (Win7 x86 Debug\Win7 x64 Release ...)
  • Tip II - Redeploying a driver to the VM after it had one successful deployment may cause problems (Such as the VM wanting to reboot).
    One possible solution would be to set a valid DriverUnload function in your driver.
    The DriverUnload function is a member of the PDRIVER_OBJECT received at DriverEntry.
    Another solution would be to return to the "Configured" snapshot before each deployment.

Summary

I hope this tip will help people new to driver development in getting started quickly.

It may seem straight forward, but I must say I encountered many problems trying to setup my environment, with nearly no useful and complete help found on the web.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Team Leader
Israel Israel
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionHad a ton of trouble setting up the machine. Pin
Eswar Yaganti25-Mar-15 4:35
Eswar Yaganti25-Mar-15 4:35 
SuggestionVisual Studio 2013 / WDK 8.1 additional setup requirements Pin
akg9213-May-14 12:35
akg9213-May-14 12:35 
QuestionI would like to add you to skype Pin
Member 1027823116-Sep-13 15:12
Member 1027823116-Sep-13 15:12 
QuestionKernel mode debugging Pin
Aster Veigas30-Aug-13 22:49
Aster Veigas30-Aug-13 22:49 
AnswerRe: Kernel mode debugging Pin
Michael Engstler30-Aug-13 23:14
Michael Engstler30-Aug-13 23:14 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:03
Aster Veigas31-Aug-13 0:03 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:12
Aster Veigas31-Aug-13 0:12 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:15
Aster Veigas31-Aug-13 0:15 
GeneralRe: Kernel mode debugging Pin
Michael Engstler31-Aug-13 3:43
Michael Engstler31-Aug-13 3:43 
I believe you must setup the VM with a named pipe, I do not know the consequences otherwise.
I suggest you follow my tutorial a step by step from the beginning (Include setting the VM with a named pipe) and try again.
I updated the tutorial with another option for deploying and debugging a VM.
I prefer the second option, and learned that it is more stable and has less problems than the first option.
GeneralRe: Kernel mode debugging Pin
Aster Veigas2-Sep-13 1:50
Aster Veigas2-Sep-13 1:50 
GeneralRe: Kernel mode debugging Pin
Michael Engstler2-Sep-13 10:07
Michael Engstler2-Sep-13 10:07 
QuestionSecond select is not necessary Pin
Member 161419123-Aug-13 0:05
Member 161419123-Aug-13 0:05 
GeneralMy vote of 4 Pin
Amit KD28-Apr-13 3:25
Amit KD28-Apr-13 3:25 
GeneralRe: My vote of 4 Pin
Aster Veigas2-Sep-13 1:51
Aster Veigas2-Sep-13 1:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.