Click here to Skip to main content
0 members
Click here to Skip to main content
Add your own
alternative version

Stats

1.1K views
1 bookmarked
Posted 1 Mar 2019
Licenced CPOL

How to Debug the Linux Kernel with QEMU and Libvirt

, , 1 Mar 2019
Rate this:
Please Sign up or sign in to vote.
And its modules during runtime

Introduction

How can you verify that your Linux kernel image will boot on real hardware? Booting your image with virtualization technologies can’t provide you with an absolute guarantee that your software will run on metal. However, QEMU and Libvirt give you enough capabilities to conduct a quick sanity check and get rid of most bugs at the development stage.

In this article, we explain how you can debug your Linux kernel and its modules during runtime. This article will be useful for Linux kernel developers who want to speed up the time to market their software.

Contents

We’ll provide you with step-by-step instructions on how you can find errors in your Linux kernel module or kernel image at the development stage. Our method includes three levels:

  • Debugging without source code
  • Debugging with source code
  • Debugging with third-party modules

You can either use one of these levels or dig deep and go through the whole process of kernel debugging.

Environment

To conduct kernel debugging, you should prepare the necessary environment. First of all, you need to set up the guest target.

The guest is a Libvirt or QEMU-style virtual machine that we’ll debug. The host is an operating system where debugging is taking place.

To create an environment, we use Ubuntu 16.04 as the host operating system and Ubuntu 16.04 as the guest operating system.

However, you can also use other guest Linux distributions. Just make sure to install the corresponding packages beforehand.

Here’s what you need to install for debugging:

  1. QEMU as a hypervisor and emulator for hardware virtualization
  2. Libvirt as a daemon and a toolkit for managing platform virtualization
  3. GDB as a project debugger
  4. A QEMU virtual machine (in our case, it’s Ubuntu 16.04 under the name 04)

Debugging Without Source Code

Debugging without the source code is the simplest way to quickly debug the kernel if you need to look at the call stack and variable values. You can verify the kernel image in this way:

  1. Configure Libvirt on the host operating system with the following command:
    virsh edit ubuntu16.04

    In your editor of choice, you can open the file with virtual machine settings. Find the domain tag and add the following option:

    <domain type='kvm'
    
            xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' >
       <qemu:commandline>
            <qemu:arg value='-s'/>
       </qemu:commandline>

    Save the file and exit the editor. Now, you need to restart the virtual machine (if it’s already running).

    The -s parameter means that QEMU will open port 1234 for debugging by default. If you need to open other ports, you can set them in the domain XML file:

    <qemu:commandline>
          <qemu:arg value='-gdb'/>
          <qemu:arg value='tcp::1235'/>
      </qemu:commandline>

    The specified port should open after the virtual machine starts.

  2. Disable Kernel Address Space Layout Randomization (KASLR) on the guest. To do this, you need to apply changes directly to the GRand Unified Bootloader (GRUB) configuration. Open /etc/default/grub and add nokaslr to the GRUB_CMDLINE_LINUX_DEFAULT variable:
    GRUB_CMDLINE_LINUX_DEFAULT="splash quiet nokaslr"

    Update GRUB with this command:

    sudo update-grub

    and reboot the guest operating system.

  3. The Linux kernel packages do not include debug symbols, as they’re stripped from binary files at build time. However, you need to install debug symbols to let GDB assist you during debugging. To install debug symbols for the existing kernel on the guest, see the detailed description here.

    Here's how we’ll install debug symbols on Ubuntu:

    #GPG key import
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
    #Add repository config
    codename=$(lsb_release -c | awk  '{print $2}')
    sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
    deb http://ddebs.ubuntu.com/ ${codename}    main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
    EOF
      
    sudo apt-get update
    sudo apt-get install linux-image-$(uname -r)-dbgsym

    Here’s how to install debug symbols on Fedora Linux: You can manually download and install the ddeb package from here as an example. However, be careful to install a package with the compatible kernel and architecture.

    sudo dnf debuginfo-instal kernel-$(uname -r)

    You can find a complete description of how to install debug symbols on Fedora Linux here.

  4. Copy the /usr/lib/debug/boot/vmlinux-$(uname -r) file from the guest to any folder on the host so GDB can use it. In our case, we have the following file: /usr/lib/debug/boot/vmlinux-4.13.0-36-generic
  5. Run the QEMU virtual machine on the host and connect it to the debugger:
    gdb ./vmlinux-4.13.0-36-generic
    (gdb) target remote :1234

    Here’s an example of Linux kernel debugging for Ubuntu 16.04 (4.13.0-36):

    kernel_new# gdb ./vmlinux-4.13.0-36-generic
    GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
    Copyright (C) 2016 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from ./vmlinux-4.13.0-36-generic...done.
    (gdb) target remote :1234
    Remote debugging using :1234
    native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
    54  /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: 
        No such file or directory.
    (gdb) bt
    #0  native_safe_halt () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
    #1  0xffffffff8191e20e in arch_safe_halt () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/paravirt.h:92
    #2  default_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:354
    #3  0xffffffff81036b45 in arch_cpu_idle () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:345
    #4  0xffffffff8191e743 in default_idle_call () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:98
    #5  0xffffffff810cd5f2 in cpuidle_idle_call () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:156
    #6  do_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:246
    #7  0xffffffff810cd853 in cpu_startup_entry (state=CPUHP_ONLINE) at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:351
    #8  0xffffffff81910e78 in rest_init () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:437
    #9  0xffffffff824a514d in start_kernel () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:702
    #10 0xffffffff824a42d5 in x86_64_start_reservations (real_mode_data= <optimized out>) 
        at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:318
    #11 0xffffffff824a4417 in x86_64_start_kernel (real_mode_data=0x8a000 
        <error: Cannot access memory at address 0x8a000>) at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:299
    #12 0xffffffff810000cf in secondary_startup_64 () at 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head_64.S:220
    #13 0x0000000000000000 in ?? ()
    (gdb) break do_dentry_open
    Breakpoint 1 at 0xffffffff812528e0: file /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c, 
        line 704.
    (gdb) c
    Continuing.
      
    Thread 1 hit Breakpoint 1, do_dentry_open (f=0xffff88007b1a4a00, 
        inode=0xffff88007a0a1008, open=0x0 <irq_stack_union>, cred=0xffff8800775eea80) 
        at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c:704
    704 /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c: No such file or directory.
    (gdb) print inode->i_ino
    $1 = 4026532024
    (gdb)

Debugging With Source Code

Source code allows the debugger to associate events and circumstances in program execution with their corresponding location. So using the source code, you can run the (gdb) list command without errors like “No such file or directory.” If you have a source code file, make it available to GDB for kernel debugging in the following way:

  1. Load the source code to the host (see detailed description here). You can read how to download the source code from the git repository here.
    git clone git://kernel.ubuntu.com/ubuntu/xenial-maverick.git
    # find for kernel 4.13.0-36
    git tag -l | grep 4.13.0-36
    # Ubuntu-hwe-4.13.0-36.40_16.04.1
    git checkout -b ubuntu-hwe-4.13.0-36 Ubuntu-hwe-4.13.0-36.40_16.04.1
  2. Connect the source code to GDB with the (gdb)set substitute-path command:
    gdb ./vmlinux-4.13.0-36-generic (gdb) set substitute-path 
        /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial
  3. If you look at the previous logs, you can see the path to the source code in the debug symbols that is /build/linux-hwe-4GXcua/linux-hwe-4.13.0.
    /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: 
        No such file or directory.

    Substitute the original source code path with the path to our source code using the set substitute-path command. In our case, the new path is /media/veracrypt2/linux_kernel_new/ubuntu-xenial.

Debugging With Third-Party Modules

This method is for those who want to debug a module that’s not included in the official Linux kernel version. For instance, we’ll try to debug our own module.

  1. Build the module with debug symbols on the guest and install it. To build the module with debug symbols, add the -g flag (EXTRA_CFLAGS += -g) to the compiler options. Copy the built module to the host folder together with the debug symbols (or to the folder with the source code). In our case, we get this:
    ls /media/veracrypt2/linux_kernel_new
    myspecificmodule.ko ubuntu-xenial vmlinux
  2. Build the GDB scripts on the host so we can debug modules that are loaded. Unfortunately, scripts are provided only with the whole kernel, so we need to stop the building process after we get the debian/build/build-generic/scripts/gdb/linux/constants.py file (you can find more information about kernel building here).
    sudo apt build-dep --yes linux-image-4.13.0-36-generic     
    sudo apt install --yes fakeroot libncurses5-dev
    fakeroot debian/rules clean
    fakeroot debian/rules binary-headers binary-generic binary-perarch

    If you want to continue building after interruption, make an additional link to the script:

    ln -s /media/veracrypt2/linux_kernel_new/ubuntu-xenial/scripts/gdb/vmlinux-gdb.py 
    /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py
  3. Permit the GDB to load. Create the ~/.gdbinit file as a user on the host:
    $ cat ~/.gdbinit
    add-auto-load-safe-path /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
    build/build-generic/vmlinux-gdb.py
  4. The file with debug symbols should be named vmlinux to make our scripts runnable, so rename this file on the host:
    mv vmlinux-4.13.0-36-generic vmlinux
  5. Run GDB and make sure that everything is okay with the scripts. We can also check if it’s possible to set breakpoints in our module and debug it (see comments in the script). The breakpoint makes the debugger stop execution after a specified function or line number in the source file.
    gdb ./vmlinux
    GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
    Copyright (C) 2016 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from ./vmlinux...done.
    (gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 
        /media/veracrypt2/linux_kernel_new/ubuntu-xenial
    (gdb) source /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
        build/build-generic/vmlinux-gdb.py 
        # load and check if scripts work correctly
    (gdb) target remote :1234
    Remote debugging using :1234
    native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54
    warning: Source file is more recent than executable.
    54  }
    (gdb) lx-symbols # load all available modules
    loading vmlinux
    scanning for modules in /media/veracrypt2/linux_kernel_new
    no module object found for 'snd_hda_codec_generic'
    no module object found for 'snd_hda_intel'
    no module object found for 'snd_hda_codec'
    no module object found for 'snd_hda_core'
    no module object found for 'snd_hwdep'
    no module object found for 'snd_pcm'
    no module object found for 'crct10dif_pclmul'
    no module object found for 'crc32_pclmul'
    no module object found for 'ghash_clmulni_intel'
    no module object found for 'pcbc'
    no module object found for 'snd_seq_midi'
    no module object found for 'snd_seq_midi_event'
    no module object found for 'aesni_intel'
    no module object found for 'aes_x86_64'
    no module object found for 'snd_rawmidi'
    no module object found for 'crypto_simd'
    no module object found for 'glue_helper'
    no module object found for 'cryptd'
    no module object found for 'snd_seq'
    no module object found for 'snd_seq_device'
    no module object found for 'snd_timer'
    no module object found for 'input_leds'
    no module object found for 'snd'
    no module object found for 'joydev'
    no module object found for 'serio_raw'
    no module object found for 'i2c_piix4'
    no module object found for 'soundcore'
    no module object found for 'mac_hid'
    no module object found for 'parport_pc'
    no module object found for 'ppdev'
    loading @0xffffffffc0151000: 
        /media/veracrypt2/linux_kernel_new/myspecificmodule.ko # our module is loaded 
    no module object found for 'lp'
    no module object found for 'parport'
    no module object found for 'autofs4'
    no module object found for 'qxl'
    no module object found for 'ttm'
    no module object found for 'drm_kms_helper'
    no module object found for 'syscopyarea'
    no module object found for 'sysfillrect'
    no module object found for 'sysimgblt'
    no module object found for 'fb_sys_fops'
    no module object found for 'psmouse'
    no module object found for 'virtio_blk'
    no module object found for 'virtio_net'
    no module object found for 'floppy'
    no module object found for 'drm'
    no module object found for 'pata_acpi'        
    (gdb) break do_dentry_open_operation_handler # breakpoint on the function of our module 
    Breakpoint 1 at 0xffffffffc0154c50: 
        file /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c, line 118.
    (gdb) c
    Continuing.
    [Switching to Thread 2]
      
    Thread 2 hit Breakpoint 1, do_dentry_open_operation_handler 
    (file=0xffff880073d2c300, inode=0xffff88007af766e8, open=0x0 <irq_stack_union>, 
        cred=0xffff8800737ff900)
        at /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c:118
    118 /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c: No such file or directory.
      
    (gdb)

    The line “no module object found for” means that there are no other modules that can be loaded. If the debugger can’t find source code files, you can manually add links to the source code for the modules. If any modules are available on the machine, you’ll see the following:

    ​​​​​​(gdb) lx-symbols # load all available modules
    loading vmlinux
    scanning for modules in /media/veracrypt2/linux_kernel_new
    loading @0xffffffffc02bb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/
      hda/snd-hda-codec-generic.ko
    loading @0xffffffffc0240000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/
      snd-hda-intel.ko
    loading @0xffffffffc0288000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/
      hda/snd-hda-codec.ko
    loading @0xffffffffc0273000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/hda/
      snd-hda-core.ko
    loading @0xffffffffc023a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-hwdep.ko
    loading @0xffffffffc0250000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-pcm.ko
    loading @0xffffffffc0235000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/
      crypto/crct10dif-pclmul.ko
    loading @0xffffffffc022b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/
      crc32-pclmul.ko
    loading @0xffffffffc026e000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/
      ghash-clmulni-intel.ko
    loading @0xffffffffc01f6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/pcbc.ko
    loading @0xffffffffc01f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/
      seq/snd-seq-midi.ko
    loading @0xffffffffc01bc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/
      seq/snd-seq-midi-event.ko
    loading @0xffffffffc01fc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aesni-intel.ko
    loading @0xffffffffc01c2000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aes-x86_64.ko
    loading @0xffffffffc01e8000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-rawmidi.ko
    loading @0xffffffffc01e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/crypto_simd.ko
    loading @0xffffffffc01b7000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/glue_helper.ko
    loading @0xffffffffc01dc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/cryptd.ko
    loading @0xffffffffc01cb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq.ko
    loading @0xffffffffc01a6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-seq-device.ko
    loading @0xffffffffc01ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-timer.ko
    loading @0xffffffffc0182000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/input-leds.ko
    loading @0xffffffffc0191000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd.ko
    loading @0xffffffffc0187000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/joydev.ko
    loading @0xffffffffc017d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/
      serio/serio_raw.ko
    loading @0xffffffffc00d3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/i2c/
      busses/i2c-piix4.ko
    loading @0xffffffffc009f000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/soundcore.ko
    loading @0xffffffffc0071000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/
      macintosh/mac_hid.ko
    loading @0xffffffffc0174000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport_pc.ko
    loading @0xffffffffc016a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/ppdev.ko
    loading @0xffffffffc0151000: /media/veracrypt2/linux_kernel_new/myspecificmodule.ko
    loading @0xffffffffc008d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/lp.ko
    loading @0xffffffffc0144000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport.ko
    loading @0xffffffffc0066000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/fs/autofs4/autofs4.ko
    loading @0xffffffffc0133000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/
      drivers/gpu/drm/qxl/qxl.ko
    loading @0xffffffffc011b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/ttm/ttm.ko
    loading @0xffffffffc00f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/
      drm/drm_kms_helper.ko
    loading @0xffffffffc0061000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/
      core/syscopyarea.ko
    loading @0xffffffffc00ea000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/
      core/sysfillrect.ko
    loading @0xffffffffc00e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/
      core/sysimgblt.ko
    loading @0xffffffffc00de000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/
      core/fb_sys_fops.ko
    loading @0xffffffffc00ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/mouse/psmouse.ko
    loading @0xffffffffc00a4000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/virtio_blk.ko
    loading @0xffffffffc0094000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/net/virtio_net.ko
    loading @0xffffffffc0079000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/floppy.ko
    loading @0xffffffffc0008000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/drm.ko
    loading @0xffffffffc0000000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/
      linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/ata/pata_acpi.ko
  6. If you want to check other modules, you can add links to the source code for them in the same way as we did in the first step:
    (gdb) set substitute-path /home/quest/Desktop/mymodule /media/veracrypt2/linux_kernel_new/mymodule

Conclusion

Debugging a Linux kernel image or module is a complex task that many developers face. Using such virtualization technologies as QEMU and Libvirt allow them to detect and fix the majority of defects at the development stage.

In this article, we showed you three levels of debugging: debugging with and without source code and debugging with the help of third-party modules. Depending on the task at hand, you can either debug your kernel image or module on one of these levels or go through the whole process.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Authors

Apriorit Inc
Chief Technology Officer Apriorit Inc.
United States United States
ApriorIT is a software research and development company specializing in cybersecurity and data management technology engineering. We work for a broad range of clients from Fortune 500 technology leaders to small innovative startups building unique solutions.

As Apriorit offers integrated research&development services for the software projects in such areas as endpoint security, network security, data security, embedded Systems, and virtualization, we have strong kernel and driver development skills, huge system programming expertise, and are reals fans of research projects.

Our specialty is reverse engineering, we apply it for security testing and security-related projects.

A separate department of Apriorit works on large-scale business SaaS solutions, handling tasks from business analysis, data architecture design, and web development to performance optimization and DevOps.

Official site: https://www.apriorit.com
Clutch profile: https://clutch.co/profile/apriorit
Group type: Organisation

33 members


Sergii Kusii
Software Developer Apriorit
Ukraine Ukraine
No Biography provided

You may also be interested in...

Pro

Comments and Discussions

 
-- There are no messages in this forum --
Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web06 | 2.8.190419.4 | Last Updated 1 Mar 2019
Article Copyright 2019 by Apriorit Inc, Sergii Kusii
Everything else Copyright © CodeProject, 1999-2019
Layout: fixed | fluid