最後更新: 2023-04-08

目錄

• blkiotune
• blkdeviotune
• Disk IO / Cache
• ksm
• CPU
• Win10
• Windows 2003

 

---

blkiotune

 

存取 device-weights

Usage:

        blkiotune <domain> [--weight <number>]

# 查看

virsh # blkiotune winxp

weight         : 500
device_weight  :

# 設定

virsh # blkiotune winxp 1000

• range [100, 1000], 0 = 去除限制

 

---

blkdeviotune

 

限制  I/O 的數量

blkdeviotune <domain> <device> [total_bytes_sec]

    --total_bytes_sec <number>  total throughput limit in bytes per second
    --read_bytes_sec <number>  read throughput limit in bytes per second
    --write_bytes_sec <number>  write throughput limit in bytes per second
    
    --total_iops_sec <number>  total I/O operations limit per second
    --read_iops_sec <number>  read I/O operations limit per second
    --write_iops_sec <number>  write I/O operations limit per second

e.g.

virsh # blkdeviotune iredmail vda

total_bytes_sec: 0
read_bytes_sec : 0
write_bytes_sec: 0
total_iops_sec : 1000
read_iops_sec  : 0
write_iops_sec : 0

P.S.

virsh # domblklist winxp

Target     Source
------------------------------------------------
vda        /mnt/vm/winxp/winxp-vio.raw
vdb        /mnt/vm/winxp/winxp-data.raw
hdc        -

domblkstat winxp vda

vda rd_req 9322
vda rd_bytes 227837440
vda wr_req 4637
vda wr_bytes 22315520
vda flush_operations 0

 

---

Disk IO / Cache

 

XML

<driver name='qemu' type='raw' cache='writethrough' io='native'/>

IO

QEMU has two asynchronous I/O mechanisms:

• threads: POSIX AIO emulation using a pool of worker threads (user-mode based threads)
• native: Linux AIO.

Setting:

... io='native'|'threads' ...

The Linux AIO mechanism has lower latency than POSIX AIO emulation

Cache

cache='writethrough'|'writeback'|'none'

• Don't use writeback unless your RAID array is battery-backed
• raw volumes or partitions, it is best to avoid the cache completely (cache='none')
• Don't use the linux filesystem btrfs on the host for the image files
• 如果係用 Windows OS, 那可以 turn off OS 內的 disk cache 去省 RAM, 因為外圍的 KVM 都有 cache 了

cache='none'

• host don't do cache
• guest disk cache is writeback
• Warn: like writeback, you can loose datas in case of a powerfailure

This mode causes qemu-kvm to interact with the disk image file or block device with O_DIRECT semantics,

so the host page cache is bypassed and I/O happens directly between the qemu-kvm userspace buffers and the storage device.

Because the actual storage device may report a write as completed

when placed in its write queue only the guest's virtual storage adapter is informed that there is a writeback cache

cache='writethrough' (defualt)

* host do read cache
* guest disk cache mode is writethrough
* writethrough make a fsync for each write.

This mode causes qemu-kvm to interact with the disk image file or block device with O_DSYNC semantics,

where writes are reported as completed only when the data has been committed to the storage device.

cache='writeback'

• host do read/write cache
• guest disk cache mode is writeback

neither O_DSYNC nor O_DIRECT semantics, so the host page cache is used and

writes are reported to the guest as completed when placed in the host page cache, and

the normal page cache management will handle commitment to the storage device.

the guest's virtual storage adapter is informed of the writeback cache,

so the guest would be expected to send down flush commands as needed to manage data integrity.

cache='directsync'

• host don't do cache.
• guest disk cache mode is writethrough

similar to writethrough, a fsync is made for each write.

qemu-kvm to interact with the disk image file or block device with both O_DSYNC and O_DIRECT semantics

it is helpful to guests that do not send flushes when needed.

cache='unsafe'

This mode is similar to the cache=writeback mode (flush commands from the guests are ignored)

Remark

"cache='none'" is probably not a good idea when you're using qcow2 files.

A qcow2 file makes it appear that every access to the disk is fragmented.

Note the cache='none' attribute that should always be specified together with io='native'

to prevent QEMU from falling back to userspace AIO.

 

---

discard & detect_zeroes

 

<driver name='qemu' type='qcow2' discard='unmap' detect_zeroes='unmap'/>

discard

attribute controls whether discard requests (also known as "trim" or "unmap") are ignored or passed to the filesystem.

"unmap" (allow the discard request to be passed)

"ignore" (ignore the discard request).

detect_zeroes

to detect zero write requests. The value can be "off", "on" or "unmap"

"unmap" turns the detection on and additionally tries to discard such areas from the image based on the value of discard above

(it will act as "on" if discard is set to "ignore")

 

---

AIO

 

aio-nr shows the current system-wide number of asynchronous io requests.

aio-max-nr allows you to change the maximum value aio-nr can grow to.

Checking

sysctl fs.aio-nr fs.aio-max-nr

fs.aio-nr = 2661
fs.aio-max-nr = 65536

Remark

native I/O needs either no disk cache or directsync cache mode, QEMU will fallback to aio=threads

<disk type='file' device='disk'>
    <driver name='qemu' type='qcow2' cache='writeback' io='native'/>  # 錯過
    ..
</disk>

 

---

IOThreads

 

The main loop and IOThreads

QEMU is an event-driven program that can do several things at once using an event loop. 

The VNC server and the QMP monitor are both processed from the same event loop,

which monitors their file descriptors until they become readable and then invokes a callback.

The optional iothread attribute assigns the disk to an IOThread as defined by the range for the domain iothreads value.

Multiple disks may be assigned to the same IOThread and are numbered from 1 to the domain iothreads value.

There should be only 1 or 2 IOThreads per host CPU

Per default, all I/O requests are handled in a single event loop within QEMU's main thread.

Using separate threads for I/O event handling can significantly improve throughput of virtual disks.

I/O threads must be allocated explicitly and disks must be associated to them.

Settings

   <domain>
      <iothreads>2</iothreads>    # 設定了啟動多次段 IO Thread
      ...
   </domain>

   <devices>
      <disk type='block' device='disk>
         # assign the disk to the I/O thread number {start from 1 ..}
         <driver name='qemu' type='raw' iothread='1'/>
         ...
      </disk>
      ...
   <devices>

 * The number of I/O threads should not exceed the number of host CPUs.

 * native I/O needs either no disk cache or directsync cache mode, QEMU will fallback to aio=threads

Checking

virsh # qemu-monitor-command win7-iv --pretty '{"execute": "query-iothreads"}'

如果是用 IOThreads

{
  "return": [
    {
      "poll-shrink": 0,
      "thread-id": 28186,
      "poll-grow": 0,
      "poll-max-ns": 32768,
      "id": "iothread3"
    },
    {
      "poll-shrink": 0,
      "thread-id": 28184,
      "poll-grow": 0,
      "poll-max-ns": 32768,
      "id": "iothread1"
    }
  ],
  "id": "libvirt-18"
}

 

---

ksm

 

# libvirt disable ksm

<memoryBacking>
 <nosharepages/>
 ...
</memoryBacking>

 * "hard_limit" must be set to the maximum memory configured for the guest
    plus any memory consumed by the process itself

# memtune

<memtune>
  <hard_limit>131072</hard_limit>
  ...
</memtune>

# hard_limit(kilobytes):             Maximum memory the guest domain can use.

# swap-hard-limit(kilobytes):       Maximum swap memory the guest domain can use.

# prevents the host from swapping out memory pages belonging to the guest

<memoryBacking>
 <locked/>
 ...
</memoryBacking>

 

---

Win10

 

[1] CPU Feature

  <cpu mode='host-model' check='partial'>
    <feature policy='disable' name='svm'/>
  </cpu>

[2] Disable hpet

  <clock offset='localtime'>
    <timer name='hpet' present='no'/>
  </clock>

[3] Use virtio driver (HDD)

0) add second virtio disk

加一隻 disk 並 mount 到先.

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/ssd/s2016/swap.qcow2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
    </disk>

1) To boot in safemode

bcdedit /set {current} safeboot minimal

2) 修改 ISO XML

# 由 sata 改成 virtio

<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='writeback'/>
  <source file='/mnt/raid/VMs/win10/win10.qcow2'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <boot order='1'/>
</disk>

# Driver ISO - virtio-win-0.1.217.iso    # 599M

<disk type='file' device='cdrom'>
  <driver name='qemu' type='raw'/>
  <source file='/mnt/raid/iso/virtio-win/virtio-win-0.1.240.iso'/>
  <target dev='hdc' bus='ide'/>
</disk>

3) 載入 Driver

經歷 2 次 "INACCESSIBLE_BOOT_DEVICE" boot fail

Troubleshoot -> Command Prompt

# 查看 DeviceID

wmic logicaldisk get deviceid, volumename, description

Description       DevicelD  VolumeName
CD-ROM Disc       D:        virtio-win-O.1.249
Local Fixed Disk  X:        Boot

# CD-ROM 係 D:

drvload D:\amd64\w10\vioscsi.inf

drvload D:\amd64\w10\viostor.inf

# 成功 load driver 後, Windows 在 "E:" (DeviceID) 出現 ^.^

Description       DevicelD  VolumeName
Local Fixed Disk  C:        System Reserved
CD-ROM Disc       D:        virtio-win-O.1.249
Local Fixed Disk  E:        OS
Local Fixed Disk  X:        Boot

# 將 Driver 打入 OS

dism /image:E:\ /add-driver /driver:D:\amd64\w10\vioscsi.inf

dism /image:E:\ /add-driver /driver:D:\amd64\w10\viostor.inf

之後 reboot 機

4) 成功 boot 起後 Disable 返 safeboot

# Once you are done you can return to normal boot by using:

bcdedit /deletevalue {current} safeboot
 

 

---

Windows 2003

 

Power settings

enable (Start -> Run -> secpol.msc -> "Security Settings\Local Policies\Security Options")
"Shutdown: Allow system to be shut down without having to log on".
"Interactive logon: Do not require CTRL+ALT+DEL"

# disable logon screen screensaver using regedit

[HKEY_USERS\.DEFAULT\Control Panel\Desktop]
"ScreenSaveActive"="0"

# Shutdown Warning Dialog Timeout

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows]
"ShutdownWarningDialogTimeout"=dword:00000001

Caching

Under disk properties in  (Windows caches writes in ram). KVM enables it's own writethough caching which relies on the Debian hosts caching

Computer Mgmt -> Disk Mgmt -> Select Drive (system) -> properties -> Hardware -> VirtIO Drv -> Properties -> Policies -> Optimize for quick removal

E1000 NIC

Use e1000 NIC. It's stable and performs well. (Windows Server 2003 R2 SP2 ships with drivers)

<model type='e1000'/>