LCE-EU 2015

Virtualization drivers (e.g. libvirt, QEMU/KVM) are the core part of OpenStack Compute ('Nova') layer. An OpenStack environment is challenging to debug as is -- more so when multiple Compute nodes and thereby multiple libvirt daemons and QEMU instances are involved. A good grasp of Virtualization debugging mechanisms is vital for effective root cause analysis. To that end, libvirt and QEMU provide a rich set of debugging controls that allow us to query (or modify) the state of virtual machines in distress.

This talk focuses on providing an in-depth view of aforementioned techniques. Topics include: debugging Nova Compute process crashes; gathering specific patterns from libvirt log filters, environment variables, and systemd journal; live querying the VM (and QEMU) state through `virsh` and QEMU QMP commands; tuning the libvirt daemon logging; monitoring events emitted by QEMU, etc.