fio

由 datahunter 在五, 18/10/2019 - 10:52 發表

最後更新: 2019-10-18

介紹

HomePage: https://github.com/axboe/fio

fio = Flexible I/O

test a specific workload (simulate a given I/O workload)

支持許多種不同的 I/O 引擎, 包括: mmap, sync ... libaio, rdma, posixaio ...

列出支援的 IO engine:

# --enghelp[=ioengine[,command]])

fio --enghelp

Available IO engines:
        cpuio
        mmap
        sync
        psync
...

Engine

sync

Basic read(2) or write(2) I/O. lseek(2) is used to position the I/O location.

See fsync and fdatasync for syncing write I/Os.

psync

Basic pread(2) or pwrite(2) I/O. Default on all supported operating systems except for Windows.

libaio

Linux native asynchronous I/O

Note that Linux may only support queued behavior with non-buffered I/O

(set `direct=1' or `buffered=0')

buffered=bool

Default: true; use buffered I/O.

This is the opposite of the direct option.

direct=bool

Default: false. If value is true, use non-buffered I/O. (O_DIRECT)

posixaio

POSIX asynchronous I/O using aio_read(3) and aio_write(3).

fsync

If writing to a file, issue an fsync(2) (or its equivalent) of the dirty data for every number of blocks given.

mmap

File is memory mapped with mmap(2) and data copied to/from using memcpy(3)

Benchmarking using fio

Install

Linux

apt-get install fio

Windows

https://ci.appveyor.com/project/axboe/fio

Job -> Artifacts

準備

echo mq-deadline > /sys/block/sdb/queue/scheduler
echo mq-deadline > /sys/block/sdc/queue/scheduler
echo 1 > /proc/sys/vm/dirty_background_ratio
echo 50 > /proc/sys/vm/dirty_ratio
echo 500 > /proc/sys/vm/dirty_expire_centisecs
/sbin/blockdev --setra 256 /dev/sdX

Opt

--readonly # extra safety guard to prevent users from accidentally starting a write or trim workload

設定

---

readwrite=str # read, write, randread, randwrite, randrw (--rwmixread=50)

---

direct=1 # Default: 0; use non-buffered I/O. This is usually O_DIRECT.

---

bs=int # block size. 單次 IO 的大小. Default: 4096.

---

size=int # The total size of file I/O for each thread of this job.

# Fio will run until this many bytes has been transferred,

# unless runtime is limited by other options (runtime)

runtime=sec # Tell fio to terminate processing after the specified period of time.

numjobs=int # Default: 1; Create the specified number of clones of this job.

# Each clone of job is spawned as an independent thread or process.

loops=int # Run the specified number of iterations of this job.

group_reporting # groups of jobs as a whole instead of for each individual job

----

iodepth=int # Default: 1; Number of I/O units to keep in flight against the file.

# iodepth beyond 1 will not affect synchronous ioengines

# 亦即是 async engines 才有 iodepth

# Even async engines may impose OS restrictions

# Keep an eye on the I/O depth distribution in the fio output to verify that the achieved depth is as expected.

# If the I/O engine is async, how large a queuing depth do we want to maintain

Read Write Performance Test

Usage

fio [options] [jobfile] ...

If the job file contains only one job, you may as well just give the parameters on the command line

Read Test

mkdir iotest; cd iotest

Basic

fio --name=MyReadTest --readwrite=read --direct=1 --size=4G

MyReadTest: (g=0): rw=read, 
            bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, 
            ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
MyReadTest: Laying out IO file (1 file / 4096MiB)  <-- 建立一個4G檔案

過程

Jobs: 1 (f=1): [R(1)][27.3%][r=41.2MiB/s][r=10.7k IOPS][eta 00m:24s]

Jobs: 1 # number of threads currently running and doing I/O

(f=?) # the number of currently open files

[R(1)]: # current status of each thread, R=sequential writes

[27.3%] # the estimated completion percentage

[w=128MiB/s] # last check bandwidth

[w=32.8k IOPS] # last check IOPS

End

MyReadTest: (groupid=0, jobs=1): err= 0: pid=4357: Thu Feb  4 15:49:39 2021
  read: IOPS=10.3k, BW=40.3MiB/s (42.2MB/s)(4096MiB/101654msec)
    clat (usec): min=84, max=39779, avg=95.62, stdev=127.63
     lat (usec): min=84, max=39779, avg=95.74, stdev=127.63
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   89], 10.00th=[   89], 20.00th=[   90],
     | 30.00th=[   90], 40.00th=[   91], 50.00th=[   92], 60.00th=[   92],
     | 70.00th=[   93], 80.00th=[   94], 90.00th=[   98], 95.00th=[  103],
     | 99.00th=[  165], 99.50th=[  184], 99.90th=[  247], 99.95th=[  379],
     | 99.99th=[ 7701]
   bw (  KiB/s): min=19248, max=43409, per=100.00%, avg=41258.42, stdev=1914.18, samples=203
   iops        : min= 4812, max=10852, avg=10314.61, stdev=478.55, samples=203
  lat (usec)   : 100=92.66%, 250=7.24%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=2.02%, sys=58.25%, ctx=1048648, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.3MiB/s (42.2MB/s), 40.3MiB/s-40.3MiB/s (42.2MB/s-42.2MB/s), 
         io=4096MiB (4295MB), run=101654-101654msec

Disk stats (read/write):
    dm-0: ios=1048378/20, merge=0/0, ticks=85720/88, in_queue=85756, util=84.24%,
          aggrios=1048576/23, aggrmerge=0/3, aggrticks=92291/131, aggrin_queue=84532, aggrutil=83.09%
  sdc: ios=1048576/23, merge=0/3, ticks=92291/131, in_queue=84532, util=83.09%

More Setting

fio --name=MyReadTest \
--readwrite=read \
--ioengine=libaio --direct=1 --iodepth=16 \
--size=4G --runtime=600\
--bs=4k \
--loops=3 --numjobs=4 --group_reporting \
--filename=/root/MyReadTest.fio

過程

fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [W(4)][44.6%][w=9537KiB/s][w=2384 IOPS][eta 02m:13s]

P.S.

會在當前目錄建立 MyTest.0.0 MyTest.1.0 MyTest.2.0 MyTest.3.0, 每個 512M

filename=str

Fio normally makes up a filename based on the job name, thread number

If you want to share files between threads in a job or several jobs with fixed file paths

you can specify a number of files by separating the names with a ':'

The size of regular files specified by this option will be "size" divided by number of files

jobfile

myjob.ini

[global]
readwrite=read
direct=1
ioengine=libaio
size=4G
write_bw_log=bw
write_iops_log=iops
# Prefix "filename" with this directory.
;directory=str
[MyReadTest]
description=job1
filename=/mydata/iotest/test.bin

fio --parse-only myjob.ini # don't start any I/O, 只 check syntax

fio --readonly myjob.ini # make sure readonly

log (feed 落 fio2gnuplot)

write_bw_log=bw       # If no str argument is given, default: jobname_type.x.log
                      # "x" is the index of the job
                      # "=foo" => foo_bw.x.log
write_iops_log=iops

randread / randwrite Test

Random Read Test

fio --name=MyTest --ioengine=libaio --direct=1 --loops=3 \
--readwrite=randread --iodepth=16 --bs=4k --size=512M \
--numjobs=4 --runtime=240 --group_reporting

Output

MyTest: (groupid=0, jobs=4): err= 0: pid=16939: Thu Oct 17 14:56:54 2019
  read: IOPS=316, BW=1268KiB/s (1298kB/s)(297MiB/240017msec)
    slat (usec): min=4, max=662603, avg=12611.44, stdev=9067.83
    clat (msec): min=9, max=1093, avg=189.23, stdev=51.90
     lat (msec): min=19, max=1128, avg=201.84, stdev=54.38
....

說明

slat Submission latency

This is the time it took to submit the I/O.

For sync I/O this row is not displayed as the slat is really the completion latency

(since queue/complete is one operation there)

clat Completion latency

time from submission to completion of the I/O pieces.

For sync I/O, clat will usually be equal (or very close) to 0

(submit to complete is basically just CPU time)

lat Total latency

the time from when fio created the I/O unit to completion of the I/O operation

Rand Write Test

瀏覽次數： 1645

夢想家

fio

介紹

Benchmarking using fio

Read Write Performance Test

randread / randwrite Test