Click here to Skip to main content
11,717,509 members (80,603 online)
Click here to Skip to main content

Kernel-Mode Debugging in a VM using Visual Studio 2012

, 31 Aug 2013 CPOL 22.7K 19
Rate this:
Please Sign up or sign in to vote.
This tip is about how to setup Kernel-Mode Debugging in a VM using Visual Studio 2012.

Introduction

This tip is about how to setup Kernel-Mode debugging in a VM using Visual Studio 2012. This tip will be helpful for people new to driver development that wish to create, build, deploy, and debug a simple "HelloKernel" project.

Background

The setup process can be divided into three parts:

  • Setting up the Virtual Machine
  • Setting up Visual Studio 2012
  • Setting up your HelloKernel project

Each part will be explained step by step.

How to Setup the Virtual Machine

The Virtual Machine setup was tested on VMWare 9.0 running Windows 7 x64 OS.

  • Create a new VM, use default settings (tested on VmWare, Win7 x64)
  • Add a serial port to the VM, follow these steps:
    • Right click VM -> Settings -> Add -> Serial Port -> Output to named pipe
    • Named pipe = \\.\pipe\com_2
    • First combo box = The end is the server
    • Second combo box = The other end is an application
    • Connect at power on = Checked
    • Finish
    • Yield CPU on poll = Checked
  • In the VM, in an elevated command prompt window, enter the following commands:
    • bcdedit /debug on
    • bcdedit /dbgsettings serial debugport:2 baudrate:115200
  • In Windows, search for "Manage advanced sharing settings":
    • Set "Turn on file and printer sharing"
    • Set "Turn off password protected sharing"
  • In Windows, search for "Computer Management":
    • Local Users and Groups -> Users:
    • Administrator -> Right Click -> Properties -> Uncheck Account is disabled
    • Administrator -> Right Click -> Set Password -> Set your password
  • Install WDK Test Target, the setup can be found on the host computer in the following path:
    "C:\Program Files (x86)\Windows Kits\8.1\Remote"
  • Reboot the VM

How to Setup Visual Studio

  • Install Visual Studio 2012 (or above) and WDK 8 (or above)
  • Run Visual Studio as administrator
  • On the main toolbar, press Driver-> Test -> Configure Computers -> Add New Computer:
    • Computer name = VM name
    • Select Provision computer and choose debugger settings
    • Next
    • Connection Type = Serial
    • Pipe = Checked
    • Reconnect = Checked
    • Port = \\.\pipe\com_2
    • Target Port = com2
    • Next -> Finish (Don’t worry if Status is Undetermined)
  • Reboot the VM
  • Create a snapshot in the VM called "Configured"
  • You may be prompted for Username\Password, enter Administrator\YourPassword
  • Important! Currently Visual Studio 2012 has a weird defect that causes the steps above to fail if your Windows username has spaces in it. The active username must have no spaces !

How to Setup Your HelloKernel Project

  • Use this tutorial to create your HelloKernel project:
  • In order to debug your project, do the following:
    • OPTION I
      • Press F6 (Important! A newly configured VM requires one successful build before debugging)
      • Press F5
      • When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • In the "Debugger Immediate Window" write the following commands:
      • bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • g (Tells the debugger to continue)
    • OPTION II
      • Open two instances of VS, one will be the Deployer and the second the Debugger
      • [In Debugger] On the main toolbar press Debug -> Attach to Process
      • [In Debugger] Select "Windows Kernel Mode Debugger" and attach to the VM
      • [In Debugger] When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • [In Debugger] In the "Debugger Immediate Window", write the following commands:
      • [In Debugger] bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • [In Debugger] g (Tells the debugger to continue)
      • [In Deployer] Press F6
    • OPTION III
      • Setup VS as a Debugger, as done in OPTION II
      • Copy the package folder located at "HelloWorld\Win7Debug\HelloWorld Package" into the VM
      • In the VM, create a new file named install.bat with the following commands:
      • sc stop HelloWorld
      • "C:\DriverTest\devcon.exe" install "C:\HelloWorld Package\HelloWorld.inf" Root\HelloWorld
      • sc start HelloWorld
      • In the VM, run install.bat as administrator
      • If prompted, press "Install this driver software anyway"
      • In order to reinstall a newer version of the driver, just update the package folder and rerun install.bat
  • Your code should break on DriverEntry, and you can start debugging

Tips

  • Tip I - Be sure your solution configuration is valid (Win7 x86 Debug\Win7 x64 Release ...)
  • Tip II - Redeploying a driver to the VM after it had one successful deployment may cause problems (Such as the VM wanting to reboot).
    One possible solution would be to set a valid DriverUnload function in your driver.
    The DriverUnload function is a member of the PDRIVER_OBJECT received at DriverEntry.
    Another solution would be to return to the "Configured" snapshot before each deployment.

Summary

I hope this tip will help people new to driver development in getting started quickly.

It may seem straight forward, but I must say I encountered many problems trying to setup my environment, with nearly no useful and complete help found on the web.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Michael Engstler
Team Leader
Israel Israel
No Biography provided

You may also be interested in...

Comments and Discussions

 
QuestionHad a ton of trouble setting up the machine. Pin
Eswar Yaganti25-Mar-15 4:35
memberEswar Yaganti25-Mar-15 4:35 
SuggestionVisual Studio 2013 / WDK 8.1 additional setup requirements Pin
abch13-May-14 12:35
memberabch13-May-14 12:35 
QuestionI would like to add you to skype Pin
Member 1027823116-Sep-13 15:12
memberMember 1027823116-Sep-13 15:12 
QuestionKernel mode debugging Pin
Aster Veigas30-Aug-13 22:49
memberAster Veigas30-Aug-13 22:49 
AnswerRe: Kernel mode debugging Pin
Michael Engstler30-Aug-13 23:14
memberMichael Engstler30-Aug-13 23:14 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:03
memberAster Veigas31-Aug-13 0:03 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:12
memberAster Veigas31-Aug-13 0:12 
GeneralRe: Kernel mode debugging Pin
Aster Veigas31-Aug-13 0:15
memberAster Veigas31-Aug-13 0:15 
GeneralRe: Kernel mode debugging Pin
Michael Engstler31-Aug-13 3:43
memberMichael Engstler31-Aug-13 3:43 
GeneralRe: Kernel mode debugging Pin
Aster Veigas2-Sep-13 1:50
memberAster Veigas2-Sep-13 1:50 
GeneralRe: Kernel mode debugging Pin
Michael Engstler2-Sep-13 10:07
memberMichael Engstler2-Sep-13 10:07 
QuestionSecond select is not necessary Pin
Member 161419123-Aug-13 0:05
memberMember 161419123-Aug-13 0:05 
GeneralMy vote of 4 Pin
Mathew D28-Apr-13 3:25
memberMathew D28-Apr-13 3:25 
GeneralRe: My vote of 4 Pin
Aster Veigas2-Sep-13 1:51
memberAster Veigas2-Sep-13 1:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.150901.1 | Last Updated 31 Aug 2013
Article Copyright 2013 by Michael Engstler
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid