FUJITSU Server PRIMERGY TX1310 M3 を買って、一つ前のブログ記事にも書いたけど、「FUJITSU Server PRIMERGY TX1310 M3」のマザーボードにも M.2 スロットがついていて、それとは別に2つの PCI Express の M.2 スロット拡張ボードを買って（1つは私が間違って買ってしまったもの）、それぞれの読み書きの速度を調査したので、ブログ記事に書いておく。

結論を先に書いておくと、どれも読み書きの速度は大きくは変わらなかったので、「FUJITSU Server PRIMERGY TX1310 M3」で M.2 SSD を使いたい場合は、マザーボードの M.2 スロットに挿して、ヒートシンクを別につけるのがいいと思う。

調査に使用した M.2 SSD

一つ前のブログ記事にも書いたけど、調査に使用したのはシリコンパワーの下記の M.2 SSD。

シリコンパワー SSD 512GB 3D NAND M.2 2280 PCIe3.0×4 NVMe1.3 P34A80シリーズ 5年保証 SP512GBP34A80M28

シリコンパワー SSD 512GB 3D NAND M.2 2280 PCIe3.0×4 NVMe1.3 P34A80シリーズ 5年保証 SP512GBP34A80M28

速度調査の方法

下記のサイトを参考に、 fio コマンドを使って調査した。

Linuxでディスク読書速度を測定（ベンチマーク）する方法 – 備忘録

「FUJITSU Server PRIMERGY TX1310 M3」のマザーボードの M.2 スロットの結果

結果部分だけ抜粋

書き込み

Run status group 0 (all jobs):
  WRITE: bw=1715MiB/s (1798MB/s), 1715MiB/s-1715MiB/s (1798MB/s-1798MB/s), io=100GiB (108GB), run=60001-60001msec

読み込み

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

「玄人志向 STANDARDシリーズ PCI-Express x4接続 M.2スロット増設インターフェースボード M.2-PCIE」の結果

玄人志向 STANDARDシリーズ PCI-Express x4接続 M.2スロット増設インターフェースボード M.2-PCIE

玄人志向 STANDARDシリーズ PCI-Express x4接続 M.2スロット増設インターフェースボード M.2-PCIE

結果部分だけ抜粋

書き込み

Run status group 0 (all jobs):
  WRITE: bw=1965MiB/s (2060MB/s), 1965MiB/s-1965MiB/s (2060MB/s-2060MB/s), io=115GiB (124GB), run=60084-60084msec

読み込み

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=19.6GiB (21.1GB), run=60001-60001msec

「アイネックス ヒートシンク搭載 M.2 SSD 変換PCIeカード AIF-08」の結果

アイネックス ヒートシンク搭載 M.2 SSD 変換PCIeカード AIF-08

アイネックス ヒートシンク搭載 M.2 SSD 変換PCIeカード AIF-08

結果部分だけ抜粋

書き込み

Run status group 0 (all jobs):
  WRITE: bw=1898MiB/s (1991MB/s), 1898MiB/s-1898MiB/s (1991MB/s-1991MB/s), io=111GiB (119GB), run=60003-60003msec

読み込み

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60002-60002msec

参考: SATA + HDD の場合

結果部分だけ抜粋

書き込み

Run status group 0 (all jobs):
  WRITE: bw=896MiB/s (939MB/s), 896MiB/s-896MiB/s (939MB/s-939MB/s), io=53.4GiB (57.3GB), run=60996-60996msec

読み込み

Run status group 0 (all jobs):
   READ: bw=965KiB/s (988kB/s), 965KiB/s-965KiB/s (988kB/s-988kB/s), io=56.6MiB (59.3MB), run=60041-60041msec
Run status group 0 (all jobs):
   READ: bw=965KiB/s (988kB/s), 965KiB/s-965KiB/s (988kB/s-988kB/s), io=56.6MiB (59.3MB), run=60041-60041msec

比較しながら見ていく

↑ のデータだけではよくわからないので、比較しながら見ていきます。

WRITE の bw

ということで、まずは WRITE の bw を見ていく。

bw は “Average bandwidth rate” 、つまり「平均の帯域幅」のことらしい。

参考: fioコマンド – 新宿区で働くプロジェクトマネージャーのブログ

マザーボード

bw=1715MiB/s (1798MB/s), 1715MiB/s-1715MiB/s (1798MB/s-1798MB/s)

玄人志向

bw=1965MiB/s (2060MB/s), 1965MiB/s-1965MiB/s (2060MB/s-2060MB/s)

アイネックス

bw=1898MiB/s (1991MB/s), 1898MiB/s-1898MiB/s (1991MB/s-1991MB/s)

SATA + HDD

bw=896MiB/s (939MB/s), 896MiB/s-896MiB/s (939MB/s-939MB/s)

bw は「玄人志向」が一番スコアが良かった。けど、数値を見る限りではそれほど差はなさそう。「玄人志向」と「SATA +HDD」との比較だと、「SATA + HDD」に比べて約2.2倍のスコアになっている。

WRITE の io と run

io は書き込んだ容量で、 run は実行時間のことだと思われるので、 io が多くて run が少ない方がスコアが良い、となると思われる。

マザーボード

io=100GiB (108GB), run=60001-60001msec

玄人志向

io=115GiB (124GB), run=60084-60084msec

アイネックス

io=111GiB (119GB), run=60003-60003msec

SATA + HDD

io=53.4GiB (57.3GB), run=60996-60996msec

run はどれもほとんど変わらないので、 io だけ見れば良さそう。 io のスコアも「玄人志向」が一番良い。「玄人志向」と「SATA +HDD」との比較だと、「SATA + HDD」に比べて約2.15倍のスコアになっている。

READ の bw

次に READ の bw を見ていく。

マザーボード

bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s)

玄人志向

bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s)

アイネックス

bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s)

SATA + HDD

bw=965KiB/s (988kB/s), 965KiB/s-965KiB/s (988kB/s-988kB/s)

これも「玄人志向」が一番スコアが良かった。けど、「マザーボード」も「アイネックス」も数MBの差なので、ほとんど変わらない気がする。「玄人志向」と「SATA + HDD」との比較だと、「SATA + HDD」に比べて約351倍（よく見ると桁の単位が違う！）のスコアになっている。

READ の io と run

次に READ の io と run を見ていく。

マザーボード

io=19.4GiB (20.8GB), run=60001-60001msec

玄人志向

io=19.6GiB (21.1GB), run=60001-60001msec

アイネックス

io=19.6GiB (21.0GB), run=60002-60002msec

SATA + HDD

io=56.6MiB (59.3MB), run=60041-60041msec

今回も run はほとんど変わらないので割愛。 io のスコアも「玄人志向」が一番良い。「玄人志向」と「SATA + HDD」との比較だと、「SATA + HDD」に比べて約350倍（これもよく見ると桁の単位が違う！）のスコアになっている。

以上、「FUJITSU Server PRIMERGY TX1310 M3」のマザーボードの M.2 スロットとその他いくつかの M.2 スロット拡張ボードのベンチマークをした話でした。

私はアイネックスを選択したけど、結果だけ見ると玄人志向の方が良かったかもしれない。（マザーボードの M.2 スロットもそんなに変わらないので、そもそもマザーボードの M.2 スロットでよかった気がする。）

以下はここまでの比較では省いた詳細のログ。参考までに。

調査ログ

「FUJITSU Server PRIMERGY TX1310 M3」のマザーボード

書き込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --filename=benchmark --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randwrite --blocksize=4k --group_reporting
benchmark: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=1060MiB/s][r=0,w=271k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=2762: Thu Sep 19 10:29:11 2019
  write: IOPS=439k, BW=1715MiB/s (1798MB/s)(100GiB/60001msec)
    clat (nsec): min=1671, max=124057k, avg=9228.78, stdev=296993.93
     lat (nsec): min=1731, max=124057k, avg=9349.00, stdev=298879.77
    clat percentiles (nsec):
     |  1.00th=[    1912],  5.00th=[    1976], 10.00th=[    2024],
     | 20.00th=[    2096], 30.00th=[    2192], 40.00th=[    2288],
     | 50.00th=[    2384], 60.00th=[    2480], 70.00th=[    2608],
     | 80.00th=[    2800], 90.00th=[    3280], 95.00th=[    4192],
     | 99.00th=[    7648], 99.50th=[   14784], 99.90th=[  317440],
     | 99.95th=[ 3751936], 99.99th=[16580608]
   bw (  KiB/s): min=  696, max=524288, per=14.81%, avg=259962.15, stdev=124446.84, samples=806
   iops        : min=  174, max=131072, avg=64990.50, stdev=31111.74, samples=806
  lat (usec)   : 2=6.66%, 4=87.63%, 10=4.97%, 20=0.33%, 50=0.20%
  lat (usec)   : 100=0.07%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=5.04%, sys=16.76%, ctx=256754, majf=0, minf=263
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,26338872,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1715MiB/s (1798MB/s), 1715MiB/s-1715MiB/s (1798MB/s-1798MB/s), io=100GiB (108GB), run=60001-60001msec

Disk stats (read/write):
    dm-0: ios=0/817894, merge=0/0, ticks=0/15466004, in_queue=15523408, util=71.06%, aggrios=0/817826, aggrmerge=0/101, aggrticks=0/15298169, aggrin_queue=15983304, aggrutil=71.27%
  nvme0n1: ios=0/817826, merge=0/101, ticks=0/15298169, in_queue=15983304, util=71.27%

読み込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randread --blocksize=4k --group_rep
orting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=352MiB/s,w=0KiB/s][r=90.0k,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=3286: Thu Sep 19 10:31:41 2019
   read: IOPS=84.8k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
    clat (usec): min=71, max=6801, avg=92.44, stdev=23.93
     lat (usec): min=71, max=6801, avg=92.55, stdev=23.94
    clat percentiles (usec):
     |  1.00th=[   75],  5.00th=[   77], 10.00th=[   78], 20.00th=[   80],
     | 30.00th=[   82], 40.00th=[   84], 50.00th=[   87], 60.00th=[   90],
     | 70.00th=[   94], 80.00th=[  103], 90.00th=[  118], 95.00th=[  129],
     | 99.00th=[  155], 99.50th=[  165], 99.90th=[  190], 99.95th=[  200],
     | 99.99th=[  265]
   bw (  KiB/s): min=28160, max=45456, per=12.49%, avg=42360.45, stdev=3671.41, samples=952
   iops        : min= 7040, max=11364, avg=10590.09, stdev=917.86, samples=952
  lat (usec)   : 100=77.34%, 250=22.65%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.42%, sys=10.25%, ctx=5113829, majf=0, minf=268
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5086373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
    dm-0: ios=5079098/14, merge=0/0, ticks=407300/0, in_queue=407871, util=100.00%, aggrios=5086373/14, aggrmerge=0/2, aggrticks=405755/0, aggrin_queue=405443, aggrutil=100.00%
  nvme0n1: ios=5086373/14, merge=0/2, ticks=405755/0, in_queue=405443, util=100.00%

玄人志向 STANDARDシリーズ PCI-Express x4接続 M.2スロット増設インターフェースボード M.2-PCIE

書き込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --filename=benchmark --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randwrite --blocksize=4k --group_reporting
benchmark: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=1622MiB/s][r=0,w=415k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=2759: Thu Sep 19 10:00:03 2019
  write: IOPS=503k, BW=1965MiB/s (2060MB/s)(115GiB/60084msec)
    clat (nsec): min=1672, max=60012k, avg=12399.41, stdev=385564.37
     lat (nsec): min=1707, max=60012k, avg=12493.63, stdev=387389.93
    clat percentiles (nsec):
     |  1.00th=[    1928],  5.00th=[    2008], 10.00th=[    2064],
     | 20.00th=[    2160], 30.00th=[    2256], 40.00th=[    2352],
     | 50.00th=[    2480], 60.00th=[    2576], 70.00th=[    2736],
     | 80.00th=[    2992], 90.00th=[    3504], 95.00th=[    4192],
     | 99.00th=[    5984], 99.50th=[    7840], 99.90th=[   72192],
     | 99.95th=[ 8978432], 99.99th=[20054016]
   bw (  KiB/s): min=71368, max=453352, per=12.51%, avg=251637.91, stdev=64558.92, samples=957
   iops        : min=17842, max=113338, avg=62909.41, stdev=16139.74, samples=957
  lat (usec)   : 2=4.08%, 4=89.80%, 10=5.76%, 20=0.16%, 50=0.09%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.04%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.40%, sys=19.19%, ctx=143317, majf=0, minf=254
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,30223410,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1965MiB/s (2060MB/s), 1965MiB/s-1965MiB/s (2060MB/s-2060MB/s), io=115GiB (124GB), run=60084-60084msec

Disk stats (read/write):
    dm-0: ios=0/1981147, merge=0/0, ticks=0/2297027, in_queue=2322267, util=35.40%, aggrios=0/1983459, aggrmerge=0/3, aggrticks=0/2293158, aggrin_queue=2419650, aggrutil=31.98%
  nvme0n1: ios=0/1983459, merge=0/3, ticks=0/2293158, in_queue=2419650, util=31.98%

読み込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randread --blocksize=4k --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=358MiB/s,w=0KiB/s][r=91.7k,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=3295: Thu Sep 19 10:05:25 2019
   read: IOPS=85.8k, BW=335MiB/s (351MB/s)(19.6GiB/60001msec)
    clat (usec): min=17, max=6864, avg=91.40, stdev=23.85
     lat (usec): min=17, max=6864, avg=91.48, stdev=23.86
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[   76], 10.00th=[   77], 20.00th=[   79],
     | 30.00th=[   81], 40.00th=[   83], 50.00th=[   86], 60.00th=[   89],
     | 70.00th=[   94], 80.00th=[  103], 90.00th=[  117], 95.00th=[  128],
     | 99.00th=[  155], 99.50th=[  165], 99.90th=[  188], 99.95th=[  202],
     | 99.99th=[  273]
   bw (  KiB/s): min=30920, max=46344, per=12.49%, avg=42863.81, stdev=3507.77, samples=953
   iops        : min= 7730, max=11586, avg=10715.94, stdev=876.95, samples=953
  lat (usec)   : 20=0.01%, 50=0.01%, 100=77.55%, 250=22.44%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.66%, sys=10.92%, ctx=5173575, majf=0, minf=269
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5146740,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=19.6GiB (21.1GB), run=60001-60001msec

Disk stats (read/write):
    dm-0: ios=5133638/10, merge=0/0, ticks=397727/0, in_queue=399432, util=100.00%, aggrios=5146741/9, aggrmerge=0/2, aggrticks=396879/0, aggrin_queue=397219, aggrutil=100.00%
  nvme0n1: ios=5146741/9, merge=0/2, ticks=396879/0, in_queue=397219, util=100.00%

アイネックス ヒートシンク搭載 M.2 SSD 変換PCIeカード AIF-08

書き込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --filename=benchmark --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randwrite --blocksize=4k --group_reporting
benchmark: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=2277MiB/s][r=0,w=583k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=2819: Thu Sep 19 08:38:28 2019
  write: IOPS=486k, BW=1898MiB/s (1991MB/s)(111GiB/60003msec)
    clat (nsec): min=1681, max=56595k, avg=12651.73, stdev=386211.56
     lat (nsec): min=1715, max=56596k, avg=12744.96, stdev=387922.40
    clat percentiles (nsec):
     |  1.00th=[    1912],  5.00th=[    1992], 10.00th=[    2064],
     | 20.00th=[    2160], 30.00th=[    2256], 40.00th=[    2352],
     | 50.00th=[    2480], 60.00th=[    2608], 70.00th=[    2768],
     | 80.00th=[    3056], 90.00th=[    3568], 95.00th=[    4256],
     | 99.00th=[    6112], 99.50th=[    8512], 99.90th=[  109056],
     | 99.95th=[ 8978432], 99.99th=[20054016]
   bw (  KiB/s): min=12888, max=415448, per=12.46%, avg=242247.27, stdev=64229.99, samples=955
   iops        : min= 3222, max=103862, avg=60561.74, stdev=16057.48, samples=955
  lat (usec)   : 2=5.40%, 4=88.06%, 10=6.12%, 20=0.20%, 50=0.10%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.04%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.24%, sys=19.30%, ctx=144201, majf=0, minf=254
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,29161859,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1898MiB/s (1991MB/s), 1898MiB/s-1898MiB/s (1991MB/s-1991MB/s), io=111GiB (119GB), run=60003-60003msec

Disk stats (read/write):
    dm-0: ios=0/2336386, merge=0/0, ticks=0/2312204, in_queue=2328211, util=39.15%, aggrios=0/2336262, aggrmerge=0/168, aggrticks=0/2296347, aggrin_queue=2448906, aggrutil=34.71%
  nvme0n1: ios=0/2336262, merge=0/168, ticks=0/2296347, in_queue=2448906, util=34.71%

読み込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randread --blocksize=4k --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=331MiB/s,w=0KiB/s][r=84.6k,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=3341: Thu Sep 19 08:40:23 2019
   read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60002msec)
    clat (usec): min=16, max=11225, avg=91.75, stdev=29.46
     lat (usec): min=16, max=11225, avg=91.84, stdev=29.47
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[   75], 10.00th=[   77], 20.00th=[   79],
     | 30.00th=[   81], 40.00th=[   83], 50.00th=[   86], 60.00th=[   89],
     | 70.00th=[   94], 80.00th=[  103], 90.00th=[  118], 95.00th=[  129],
     | 99.00th=[  157], 99.50th=[  167], 99.90th=[  190], 99.95th=[  204],
     | 99.99th=[  302]
   bw (  KiB/s): min=27824, max=46408, per=12.50%, avg=42761.06, stdev=3687.69, samples=953
   iops        : min= 6956, max=11602, avg=10690.23, stdev=921.92, samples=953
  lat (usec)   : 20=0.01%, 50=0.01%, 100=77.00%, 250=22.99%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=2.73%, sys=10.85%, ctx=5157110, majf=0, minf=273
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5131450,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60002-60002msec

Disk stats (read/write):
    dm-0: ios=5120717/12, merge=0/0, ticks=397052/0, in_queue=398469, util=99.92%, aggrios=5131450/11, aggrmerge=0/2, aggrticks=396178/0, aggrin_queue=396475, aggrutil=100.00%
  nvme0n1: ios=5131450/11, merge=0/2, ticks=396178/0, in_queue=396475, util=100.00%

SATA + HDD

書き込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --filename=benchmark --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randwrite --blocksize=4k --group_reporting
benchmark: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 1 (f=1): [_(4),w(1),_(3)][100.0%][r=0KiB/s,w=90.9MiB/s][r=0,w=23.3k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=3368: Thu Sep 19 08:45:37 2019
  write: IOPS=229k, BW=896MiB/s (939MB/s)(53.4GiB/60996msec)
    clat (usec): min=2, max=422247, avg=28.91, stdev=1459.82
     lat (usec): min=2, max=422247, avg=29.03, stdev=1460.67
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[   11], 95.00th=[   11],
     | 99.00th=[   13], 99.50th=[   16], 99.90th=[ 6456], 99.95th=[13698],
     | 99.99th=[23987]
   bw (  KiB/s): min= 1968, max=299096, per=13.79%, avg=126464.56, stdev=66871.00, samples=883
   iops        : min=  492, max=74774, avg=31616.07, stdev=16717.76, samples=883
  lat (usec)   : 4=38.52%, 10=51.51%, 20=9.72%, 50=0.09%, 100=0.02%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=0.06%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=3.05%, sys=14.49%, ctx=83906, majf=0, minf=256
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13985581,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=896MiB/s (939MB/s), 896MiB/s-896MiB/s (939MB/s-939MB/s), io=53.4GiB (57.3GB), run=60996-60996msec

Disk stats (read/write):
  sda: ios=0/7438, merge=0/1353, ticks=0/7635466, in_queue=7725767, util=98.96%

読み込み

$ fio --time_based --name=benchmark --size=256M --runtime=60 --randrepeat=1 --iodepth=32 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=8 --rw=randread --blocksize=4k --group_reporting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
...
fio-3.7
Starting 8 processes
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
benchmark: Laying out IO file (1 file / 256MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=944KiB/s,w=0KiB/s][r=236,w=0 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=8): err= 0: pid=3389: Thu Sep 19 08:49:04 2019
   read: IOPS=241, BW=965KiB/s (988kB/s)(56.6MiB/60041msec)
    clat (msec): min=2, max=400, avg=33.14, stdev=32.67
     lat (msec): min=2, max=400, avg=33.14, stdev=32.67
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[   11],
     | 30.00th=[   13], 40.00th=[   17], 50.00th=[   22], 60.00th=[   28],
     | 70.00th=[   37], 80.00th=[   52], 90.00th=[   75], 95.00th=[  100],
     | 99.00th=[  157], 99.50th=[  180], 99.90th=[  249], 99.95th=[  275],
     | 99.99th=[  334]
   bw (  KiB/s): min=   16, max=  240, per=12.50%, avg=120.59, stdev=32.55, samples=960
   iops        : min=    4, max=   60, avg=30.09, stdev= 8.14, samples=960
  lat (msec)   : 4=0.12%, 10=18.55%, 20=29.00%, 50=31.80%, 100=15.60%
  lat (msec)   : 250=4.83%, 500=0.10%
  cpu          : usr=0.04%, sys=0.15%, ctx=14509, majf=0, minf=269
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=14487,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=965KiB/s (988kB/s), 965KiB/s-965KiB/s (988kB/s-988kB/s), io=56.6MiB (59.3MB), run=60041-60041msec

Disk stats (read/write):
  sda: ios=14485/5, merge=0/3, ticks=479102/470, in_queue=479701, util=99.96%