介紹

 

The host(Debian/RHEL) , which we call L0,

the guest hypervisor(KVM), which we call L1,

and its nested guest, which we call L2.

* Only 64-bit guest hypervisors are supported.

Why Nested Virt?

• User-controlled hypervisors
• OpenStack in a VM
• Live-migration of hypervisor

Bottleneck:

• Performance overhead {Too many VM Exits & VM Entries)

目錄

• AMD
• Intel

---

AMD

 

Host support check:

egrep -e '(vmx|svm|ept)' /proc/cpuinfo

flags           : .... svm ... svm_lock

modinfo kvm_amd | grep -i nested

parm:           nested:int

cat /sys/module/kvm_amd/parameters/nested

1

# add the cpu attribute to the regular-guest’s libvirt xml to expose AMD’s svm instructions

libvirt setting

Host:

virsh capabilities | less

<cpu mode='host-model' check='partial'/>

kvm parameter

-cpu host

        OR

-cpu qemu64,+vmx

Libvrit XML

<cpu mode='custom' match='exact' check='full'>
    <feature policy='require' name='svm'/>
    ...    
</cpu>

Guest check

egrep --color -i "svm|vmx" /proc/cpuinfo

在 Guest 上啟用 kvm

file /dev/kvm

沒有的話, 就建立它

mknod  /dev/kvm  c 10 232

chown root.kvm /dev/kvm

chmod 660 /dev/kvm

 

---

Intel

 

Host Testing

cat /sys/module/kvm_intel/parameters/nested

N

=======================================

Enable nest kvm

如果以上 OUTPUT 是 'N'

lsmod | kvm_intel

echo "options kvm-intel nested=1" | tee /etc/modprobe.d/kvm_intel.conf

rmmod kvm_intel

modprobe kvm_intel

=======================================

VM XML

enable Intel VMX(Virtual Machine Extensions) in the regular guest’s CPU.

<cpu match='exact'>
  <model>core2duo</model>
 <feature policy='require' name='vmx'/>
</cpu>

=======================================

Checking:

Host:

ps aux | grep kvm | grep --colour vmx

-cpu core2duo,+vmx

Guest:

modprobe kvm

modprobe kvm_intel

 

---

Testing

 

安裝

yum install qemu-kvm-tools

測試

kvm_stat

 efer_reload                  0       0
 exits                        0       0
 fpu_reload                   0       0
 halt_exits                   0       0
 halt_wakeup                  0       0
 host_state_reload            0       0
 hypercalls                   0       0
 insn_emulation               0       0
 insn_emulation_fail          0       0
 invlpg                       0       0
 io_exits                     0       0
 irq_exits                    0       0
 irq_injections               0       0
 irq_window                   0       0
 largepages                   0       0
 mmio_exits                   0       0
 mmu_cache_miss               0       0
 mmu_flooded                  0       0
 mmu_pde_zapped               0       0
 mmu_pte_updated              0       0
 mmu_pte_write                0       0
 mmu_recycled                 0       0
 mmu_shadow_zapped            0       0
 mmu_unsync                   0       0
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed                     0       0
 pf_guest                     0       0
 remote_tlb_flush             0       0
 request_irq                  0       0
 signal_exits                 0       0
 tlb_flush                    0       0

 

---

Nested ESXi

 

Minimum hardware configurations supported by ESXi6.7

• At least two CPU cores.
• A minimum of 4 GB of physical RAM.

Config

 <memory unit='KiB'>4194304</memory>
 <vcpu placement='static'>2</vcpu>
 <model type='e1000'/>

 

---

Performance

 

VMCS(Virtual Machine Control Structure) Shadowing (Intel)

Processor optimization that lets L0 define a shadow VMCS
Guest hypervisor can access shadow VMCS directly (in hardware)
Consequently, reducing the number of VM entries & exits (VMREAD and VMWRITE)

EPT (Extended Page Tables)

Hardware optimization that provides a second level address translation.

• Nested EPT (Intel)
• Emulate EPT for L1

Test:

MMU Combination     L2      L1      L0
---------------------------------------------
Shadow-on-Shadow    55m54s  12m27s  2m14s
Shadow-on-EPT       11m39s  2m22s   2m14s
EPT-on-EPT (nEPT)   4m53s   3m56s   2m14

oos_shadow

http://blog.xen.org/index.php/2008/08/27/xen-33-feature-shadow-3/

- kvm.oos_shadow= [KVM] Disable out-of-sync shadow paging.

- Default is 1 (enabled)

Hardware

Enhanced Virtualization (VMCS) <-- 4th gen Processors

i5-2500K <== 2nd gen

`nested`, `enable_shadow_vmcs`, `enable_apicv`, `ept`

# nested

$ cat /sys/module/kvm_intel/parameters/nested

Y

# shadow VMCS

$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs

Y

# APIC Virtualization

$ cat /sys/module/kvm_intel/parameters/enable_apicv

N

# EPT

$ cat /sys/module/kvm_intel/parameters/ept

Y

check cpu:

http://datahunter.org/msr-tools

 

---

ept: Intel Extended Page Table