![首頁 Logo Keith ]p !!](/themes/mytheme/logo.png){kind=logo}
最後更新: 2019-10-05
介紹
libvirt 是個一次過攬定 QEMU, KVM, XEN, OpenVZ, LXC, 及VirtualBox 的 VM 管理工具來
只要學一次, 日後就可用在不同的 VM 技術上 !! 所以物超所值~
不過, 我最後都是忠於原生工具 ...... 原因 ?? 因為想學得專一些 ~
目錄
- Check Hardware Support
- 安裝
- libvirt-bin
- Virsh
- XML
- VM Storage 的 XML (HardDisk, CD-ROM)
- Virtio
- Storage Pools & Storage Volumes
- video
- Input Device (mouse & tablet)
- CPU Model
- memory
- Controllers
- Network Settings
- 心得
- GUI Tools
- Troubleshoot
Check Hardware Support
CPU 支援 emulator 的能用
egrep '(vmx|svm)' --color=always /proc/cpuinfo
設定用什麼 emulator
# Ubuntu
<emulator>/usr/bin/kvm</emulator>
# Centos
<emulator>/usr/libexec/qemu-kvm</emulator>
Start guest fail
log
error: unsupported configuration: Domain requires KVM, but it is not available. Check that virtualization is enabled in the host BIOS, and host configuration is setup to load the kvm modules.
modprobe kvm-intel
FATAL: Error inserting kvm_intel (/lib/modules/2.6.32-696.1.1.el6.x86_64/kernel/arch/x86/kvm/kvm-intel.ko): Operation not supported
dmesg
dmesg | grep kvm
kvm: disabled by bios
安裝
主要工具 libvirt-bin
Ubuntu 14.04
apt-get install libvirt-bin
Centos 7
yum install qemu qemu-kvm qemu-system-x86 libvirt-daemon libvirt-client bridge-utils
lsmod | grep kvm
kvm_intel 138567 0 kvm 441119 1 kvm_intel
systemctl start libvirtd; systemctl enable libvirtd
echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/99-ipforward.conf; sysctl -p
Package:
# qemu-user-2.0.0-1.el7.6.x86_64
# QEMU user mode emulation of qemu targets
qemu-x86_64
# qemu-system-x86-2.0.0-1.el7.6.x86_64
# QEMU system emulator for x86
qemu-system-x86_64
Centos 8
yum install @virt
systemctl start libvirtd; systemctl enable libvirtd
echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/99-ipforward.conf; sysctl -p
Remark: XML
<emulator>/usr/bin/kvm</emulator> # U18 <emulator>/usr/libexec/qemu-system-x86_64</emulator> # C7 <emulator>/usr/libexec/qemu-kvm</emulator> # C8
測試
VZserver:~#virsh --version
0.8.3
VZserver:~#libvirtd --version
libvirtd (libvirt) 0.8.3
libvirt-bin
libvirt-bin 概略
/etc/default/libvirt-bin <---控制開機是否執行
start_libvirtd="yes" libvirtd_opts="-d" <-- 加多個 "-l" 就可以用 TCP 來 remote Control
/etc/init.d/libvirt-bin <-- daemon 的 on / off 控制
/usr/sbin/libvirtd <-- management daemon
設定檔的位於:
/etc/libvirt/
內有目錄:
- hooks
- nwfilter
- qemu <-- VM_Name.xml 及 networks 的設定目錄
設定:
- libvirt.conf <-- 存放 VM 的 URI aliases
- libvirtd.conf
- lxc.conf <-- 不用 VM 技術的設定
- qemu.conf <-- QEMU driver 的主設定, VMC, SPICE
Setting
# U 18
/etc/libvirt/qemu.conf
vnc_listen = "0.0.0.0" # 沒有 per-domain 密碼時會用它 # nc_password = "XYZ12345" # port for creating both VNC and SPICE sessions #remote_display_port_min = 5900 #remote_display_port_max = 65535 # The user for QEMU processes run by the system instance. user = "root" group = "root" # Whether libvirt should dynamically change file ownership # to match the configured user/group above. Defaults to 1. dynamic_ownership = 1 # mac_filter enables MAC addressed based filtering on bridge ports. # This currently requires ebtables to be installed. # #mac_filter = 1
systemctl restart libvirtd
Virsh
詳見: http://datahunter.org/kvm_virsh
XML
XML 的定義在
/usr/share/libvirt/schemas
i.e.
domain.rng
Verify XML
virt-xml-validate win7-iv.xml
win7-iv.xml validates
設定 CPU & RAM
<domain type='kvm'> ... <vcpu>2</vcpu> <memory>262144</memory> <-- kilobytes <currentMemory>262144</currentMemory> <-- actual allocation of memory (ballooning up ) 可以省略 ... </domain>
設定 Disk:
<devices>
...................
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/vm/debian1/debian1_sata-2g.raw'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
...................
</devices>
vda: 在 Guest 內的 device name
driver:
namehypervisor driver type
kvm 的 "qemu" 支持的 image format: "raw", "qcow2", and "qed"
設定 Boot 次序:
方法1 與方法2不能共存
XML 方式1:
<OS>
<boot dev='hd'/>
<boot dev='cdrom'/>
<bootmenu enable='yes'/>
</OS>
XML 方式2:
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/vm/winxp/winxp-vio.raw'/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/vm/winxp/winxp-data.raw'/>
<target dev='vdb' bus='virtio'/>
<boot order='2'/>
</disk>XML 方式3:
boot from PXE ( 它是用 ipxe(https://ipxe.org/) )
<os>
<boot dev='network'/>
...
</os>Features:
它們都是一些 CPU 功能來
<features>
<acpi/>
<apic/>
<pae/>
</features>
查看系統(Host)能力
virsh capabilities | less
<host> <cpu> ... <feature name='vme'/> <pages unit='KiB' size='4'/> <pages unit='KiB' size='2048'/> <pages unit='KiB' size='1048576'/> </cpu> ... <migration_features> <live/> <uri_transports> <uri_transport>tcp</uri_transport> <uri_transport>rdma</uri_transport> </uri_transports> </migration_features> <secmodel> <model>apparmor</model> <doi>0</doi> </secmodel> ... </host> <guest> <os_type>hvm</os_type> <arch name='i686'> ... </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> ... </guest>
Link: CPU Features
虛擬機的 type
hvm:
- <arch>x86_64</arch> # 64 bit
- <arch name="i686"></arch> # 32 bit
- ...
hvm XML:
<os>
<type arch='x86_64' machine='pc-1.1'>hvm</type>
...
</os>P.S.
64bit 的 Guest 要有 hvm 設定 !!
VM Storage 的XML
- HardDisk
- CD-ROM
IDE:
<devices>
.....
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/mnt/vm/winxp/winxp.sp3-ide8g.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>
.....
</devices>
# 它會自動加上去
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
image file 的 type 有 iso, qcow2, vmdk(不建議用, 很慢), cloop, raw(不可以 take snapshot)
SATA:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/home/vm/server03.qcow2'/>
<target dev='sda' bus='sata'/>
</disk>Floppy
<disk type='file' device='floppy'> <source file='/home/iso/virtio-win-0.1.185_amd64.vfd'/> <target dev='fda' bus='fdc'/> </disk>
* 造裡的 fdc 係有特別意思
CD-ROM:
XML:
NO DICS:
<disk type='file' device='cdrom'>
<target dev='hdc' bus='ide'/>
<readonly/>
</disk>save 後會自動變成:
<disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='0'/> </disk>
ISO:
<disk type='file' device='cdrom'> <source file='/some/path/cdimage.iso'/> <target dev='hdc' bus='ide'/> </disk>
之後會變成
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/some/path/cdimage.iso'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' unit='0'/>
</disk>
在 active domain 上入 iso:
方法1: attach-disk
attach:
virsh # attach-disk debian3 /kvm/iso/systemrescuecd-x86-2.4.0.iso hda --type cdrom
Disk attached successfully
de-attach:
virsh # attach-disk debian3 "" hda --type cdrom
Disk attached successfully
P.S. 必須加 "--type cdrom"
attach-disk win2000 "" hdc
error: missing source information for device hdc
方法2: qemu-monitor-command
在 virsh 上使用 qemu-monitor 的指令
qemu-monitor-command [--hmp] <domain> <command>
default QMP format
--hmp:
Example:
找出 VM的 CD-ROM:
virsh # domblklist s2008
Target Source ------------------------------------------------ vda /data/vm/s08r2/s08r2.qcow2 hdc -
OR
virsh # qemu-monitor-command --hmp debian3 "info block"
drive-ide0-0-0: removable=1 locked=0 tray-open=0 io-status=ok [not inserted]
入碟:
attach-disk s2008 /home/iso/virtio-win-0.1.102.iso hdc --type cdrom
OR
virsh # qemu-monitor-command --hmp debian3 "change drive-ide0-0-0 /path/to/iso"
查看:
virsh # domblklist s2008
Target Source ------------------------------------------------ vda /data/vm/s08r2/s08r2.qcow2 hdc /home/iso/virtio-win-0.1.102.iso
彈碟:
virsh # qemu-monitor-command --hmp debian3 "eject drive-ide0-0-0"
有機會被 block
[1236385.504406] type=1400 audit(1360856193.076:72): apparmor="DENIED" operation="open" parent=1 profile="libvirt-d1775aad-3151-7005-f5e3-346242c86e86" name="/mnt/vm/iso/systemrescuecd-x86-2.4.0.iso" pid=3481 comm="kvm" requested_mask="r" denied_mask="r" fsuid=108 ouid=108
change-media
彈出 iso 檔
# 查看 device 的名稱
virsh domblklist openmediavault
Target Source
-------------------------------------------------
vda /ssd/openmediavault/vda.qcow2
hdc /home/iso/systemrescue-8.03-amd64.iso# 彈出 hda (Default: --live, 另有 --config)
virsh change-media openmediavault hdc --eject
換 iso
virsh change-media openmediavault hdc --insert \
--source /home/iso/systemrescue-8.03-amd64.iso
Troubleshoot
Error
"Booting from DVD/CD... Boot failed: Could not read from CDROM" 0003
XML
<target dev='hda' bus='ide' tray='open'/>
Virtio
詳見: http://datahunter.org/kvm_virtio
Disk
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/mnt/vm/winxp-vio.qcow2'/>
<target dev='vda' bus='virtio'/>
</disk>
Net
<interface type='bridge'>
<mac address='52:54:00:d9:e4:1c'/>
<source bridge='br0'/>
<target dev='win7'/>
<model type='virtio'/>
</interface>
Storage Pools & Storage Volumes
# File
<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/dev/sdc1'/> <target dev='vda' bus='virtio'/> </disk>
type: "file", "block", "dir", "network", "volume" (refer to the underlying source for the disk)
device: "floppy", "disk"(Default), "cdrom", and "lun"
# Device
<disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/sdc'/> <target dev='sdc' bus='sata'/> </disk>
video
XML:
<video>
<model type='cirrus' vram='9216' heads='1'>
<acceleration accel3d='yes' accel2d='yes'/>
</model>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
model
type: 有 vga, cirrus, vmvga, xen, vbox, qxl
vram: 單位 bytes
Input Device (mouse & tablet)
mouse 與 tablel 是可以同時存在的
mouse:
<devices>
<input type='mouse' bus='ps2'/> // bus 另有 usb
</devices>
tablet:
<input type='tablet' bus='usb'/> // 一定是要 usb
tablet 可以解決 VNC 上 cursor 不準的問題
tablet 在 S208r2 上沒反應
Control Panel > Devices and Printers
There you can see the "QEMU USB Tablet" which is a tablet/mouse device used for the VNC-mouse to work on the guest.
Select "QEMU USB Tablet", Press Enter BTN > Hardware tab > "USB Input Device" > Properties > "Power Management" tab
Unchecking "Allow the computer to turn off this device to save power"
CPU Model
KVM 的 Guest 的 Default CPU Model
Intel:
vendor_id : GenuineIntel model name : QEMU Virtual CPU version 1.0
AMD:
vendor_id : AuthenticAMD model name : QEMU Virtual CPU version 1.0
XML
<cpu match='exact' check='partial'>
<cpu mode='host-model'/> // host-passthrough: cannot be reproduced on different hardware
// custom(default): copying host CPU definition from capabilities XML
<model fallback='allow'>coreduo</model> // fallsback to a closest model (default)
// 486, coreduo, core2duo, phenom, athlon,
// qemu32(pentiumpro + pni), qemu64 GenuineIntel, AuthenticAMD
<vendor>Intel</vendor> // 在 /usr/share/libvirt/cpu_map.xml 定義了
<topology sockets='1' cores='2' threads='1'/> // number of cores per socket,
// number of threads per core
<feature policy='disable' name='lahf_lm'/>
</cpu>
match
exact (default)
The virtual CPU provided to the guest should exactly match the specification.
If such CPU is not supported, libvirt will refuse to start the domain.
minimum
The specified CPU model and features describes the minimum requested CPU.
A better CPU will be provided to the guest if it is possible with the requested hypervisor on the current host.
check
partial
Libvirt will check the guest CPU specification before starting a domain,
but the rest is left on the hypervisor. It can still provide a different virtual CPU.
full
The virtual CPU created by the hypervisor will be checked against the CPU specification and
the domain will not be started unless the two CPUs match.
mode - CPU 'host-model', 'host-passthrough' 與 "custom"
host-passthrough
The CPU visible to the guest should be exactly the same as the host CPU
even in the aspects that libvirt does not understand.
<cpu mode='host-passthrough'>
<feature name="svm" policy="disable"/>
...
</cpu>* Using the feature element, specific flags may be enabled or disabled specifically in addition to the host model.
host-model
A shortcut to copying host CPU definition from capabilities XML into domain XML.
(Copy just before starting a domain)
<cpu mode='host-model'>
<feature name="svm" policy="disable"/> # 必須 !
...
</cpu>* exactly the same XML can be used on different hosts while still providing the best guest CPU each host supports
custom
This is the default when no mode attribute is specified.
<cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>qemu64</model> <feature policy='disable' name='svm'/> <feature policy='require' name='sse4.1'/> <feature policy='require' name='sse4.2'/> </cpu>
feature - CPU Feature(flag)
The virtual CPU will claim the feature is supported regardless of it being supported by host CPU.
require
Guest creation will fail unless the feature is supported by the host CPU or the hypervisor is able to emulate it.
optional
The feature will be supported by virtual CPU if and only if it is supported by host CPU.
disable
The feature will not be supported by virtual CPU.
forbid
Guest creation will fail if the feature is supported by host CPU.
Named model with feature customization:
找出 cpu_map.xml
# U16
dpkg -L libvirt-bin | grep cpu_map.xml
ls /usr/share/libvirt/cpu_map
index.xml x86_EPYC-IBRS.xml x86_Opteron_G4.xml ppc64_POWER6.xml x86_EPYC.xml x86_Opteron_G5.xml ...
Named model
<cpu mode='custom'>
<model>Westmere</model>
<feature name="pcid" policy="require"/>
...
</cpu>
CPU 數量及 pinning
cputune
<cputune>
<vcpupin vcpu="0" cpuset="1-4,^2"/> # default pin 到所有 CPU 上
<vcpupin vcpu="1" cpuset="0,1"/>
</cputune>vcpu 數量及分佈
<vcpu placement='static' cpuset="1-4,^3,6" current="1">2</vcpu>
cpuset
list of physical CPU numbers that domain process and virtual CPUs can be pinned to by default.
+ If "cputune->vcpupin" is specified, the cpuset specified by vcpu here will be ignored.
current
The optional attribute current can be used to specify whether fewer than the maximum number of virtual CPUs should be enabled.
placement
The value can be either "static" or "auto"
"auto", the domain process will be pinned to the advisory nodeset from querying numad and
the value of attribute cpuset will be ignored and use the settings specified in the <numatune> tag
"static", the system will use the settings specified in the <vcpu placement> tag instead of the settings in <numatune>
* CPUs can be specified separately by using the cputune attribute.
memory
<memory unit='MiB'>1024</memory> # default unit: kibibytes,
# 可以改用 unit='M' ("M" 與 "MiB" 是一樣的)
<currentMemory unit='MiB'>700</currentMemory> # ballooning 後可用的 Ram
Controllers
一般而言, 我們不用設備 Controller 的, 因為 libvirt 會為我們自動加上
設置 USB:
<controller type='usb' index='0' model='piix4-uhci'> // 另有 ich9-uhci1, ich9-uhci2
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> // 自動加上
</controller>
piix4-uhci = Intel 82371
Network Settings
List Nic Type
qemu v3.1
qemu-system-x86_64 -net nic,model=?
qemu: Supported NIC models: e1000...e1000e
e1000 vs e1000e
e1000
- emulate a 1 Gbit Intel 82540EM card
- Linux driver: e1000
e1000e
- emulate a 1 Gbit Intel 82574L card
- Linux driver: e1000e
XML
<interface type='bridge'> # libvirt 會自動生成並保存 <mac address='52:54:31:34:00:01'/> <source bridge='drdb_sw'/> <target dev='drdb-a-eth1'/> <model type='virtio'/> # 使用 jumbo frame <mtu size='9000'/> # <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface>
心得
- 當有兩個 VM 用同一個 VM Image Start 後, 那 IMAGE 會 Currupt !!
GUI Tools
- virt-viewer
- virt-manager
手機 app:
- VM Manager
libvirt VM / Domain management over SSH
https://play.google.com/store/apps/details?id=vm.manager
Troubleshoot
[1] kvm 沒有 x86_64
virsh start VM
error: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
原因: kernel module wasn 't loaded ( disabled in the bios )
verify
egrep --color '(vmx|svm)' /proc/cpuinfo
lsmod | grep kvm
virsh capabilities
