Suspicious of my previous tests using iozone, I wrote a small program (that follows) to measure synchronous write speed. I tested this with and without an SLOG device, and then used a really fast Plextor M5 Pro 256GB SSD as the SLOG. I wanted to make sure that something could achieve the fast SLOG speed that I hoped for. (Remember I am doing this in the interest of spending money on a new SSD.)
Summary
An SLOG device can degrade performance, by forcing synchronous writes to wait until they are committed to the SLOG. However, it cannot enhance performance. The max (steady-state) performance is determined by bandwidth to the main storage.
No SLOG
The benchmark shows:
16777216000 bytes written in 88.68440651893616 seconds 189.17887212130893 MB/s
A typical iostat looks like:
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.43T 1.45T 0 1.36K 0 175M
raidz1 6.43T 1.45T 0 1.36K 0 175M
gpt/TOSH-3TB-A – – 0 694 0 86.2M
gpt/TOSH-3TB-B – – 0 704 0 87.6M
gpt/SGT-USB-3TB – – 0 695 0 86.5M
cache – – – – – –
gpt/tank_cache0 174G 256M 0 927 0 116M
——————- —– —– —– —– —– —–
32GB SATA II Kingston SSD as SLOG
Python benchmark:
335544320 bytes written in 13.72151803970337 seconds 24.453877408395968 MB/s
I also tried running the native device (no gpt layer), but the results were in fact lower (maybe just a coincidence):
335544320 bytes written in 22.335944175720215 seconds 15.02261634252945 MB/s
Typical iostat:
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.42T 1.46T 0 187 0 12.4M
raidz1 6.42T 1.46T 0 0 0 0
gpt/TOSH-3TB-A – – 0 0 0 0
gpt/TOSH-3TB-B – – 0 0 0 0
gpt/SGT-USB-3TB – – 0 0 0 0
logs – – – – – –
gpt/tank_zil1 0 29.8G 0 187 0 12.4M
cache – – – – – –
gpt/tank_cache0 174G 256M 0 0 0 16.0K
——————- —– —– —– —– —– —–
and
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.43T 1.44T 0 332 0 27.2M
raidz1 6.43T 1.44T 0 144 0 14.8M
gpt/TOSH-3TB-A – – 0 67 0 7.41M
gpt/TOSH-3TB-B – – 0 67 0 7.42M
gpt/SGT-USB-3TB – – 0 66 0 7.41M
logs – – – – – –
ada2 424K 29.7G 0 187 0 12.4M
cache – – – – – –
gpt/tank_cache0 174G 256M 0 117 0 14.7M
——————- —– —– —– —– —– —–
32 GB 4kB aligned as SLOG
I thought maybe the fact that this drive identifies itself as a 512-byte sector (when in fact it is a 4k drive) was causing lower performance. So, I forced 4k sectors using gnop.
Not much different (aligned via gnop):
335544320 bytes written in 14.291123151779175 seconds 23.47921268582913 MB/s
iostat:
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.43T 1.44T 0 307 0 22.4M
raidz1 6.43T 1.44T 0 87 0 7.82M
gpt/TOSH-3TB-A – – 0 34 0 3.32M
gpt/TOSH-3TB-B – – 0 35 0 3.44M
gpt/SGT-USB-3TB – – 0 39 0 3.93M
logs – – – – – –
ada2.nop 320M 29.4G 0 220 0 14.6M
cache – – – – – –
gpt/tank_cache0 174G 256M 0 146 0 18.2M
——————- —– —– —– —– —– —–
Plextor 256GB as SLOG:
16777216000 bytes written in 83.31868481636047 seconds 201.3619878539612 MB/s
iostat:
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.45T 1.42T 4 4.46K 531K 383M
raidz1 6.45T 1.42T 4 1.63K 531K 191M
gpt/TOSH-3TB-A – – 3 799 224K 95.3M
gpt/TOSH-3TB-B – – 1 799 125K 95.5M
gpt/SGT-USB-3TB – – 2 791 176K 94.6M
logs – – – – – –
gpt/tank_cache0 1.88G 172G 0 2.83K 0 192M
——————- —– —– —– —– —– —–
No SLOG / no L2ARC:
This is the control (in the scientific method) for the case where I used the Plextor as the SLOG device. Since the plextor was the L2ARC before, I had really made two changes in making it the log:
- Removed it as L2ARC (cache)
- Added it as SLOG (ZIL)
This benchmark configuration isolates the effect of #1 without #2.
16777216000 bytes written in 87.0151207447052 seconds 192.80805285811064 MB/s
iostat:
capacity operations bandwidth
pool alloc free read write read write
——————- —– —– —– —– —– —–
tank 6.46T 1.41T 0 1.61K 51.2K 198M
raidz1 6.46T 1.41T 0 1.61K 51.2K 198M
gpt/TOSH-3TB-A – – 0 806 19.2K 98.4M
gpt/TOSH-3TB-B – – 0 811 6.40K 99.0M
gpt/SGT-USB-3TB – – 0 806 19.2K 99.4M
——————- —– —– —– —– —– —–
Synchronous write benchmark in Python
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
import os import random import argparse import logging import time if __name__ == '__main__': logging.basicConfig(filename='disk_test.log', format="%(asctime)s - %(message)s", level=logging.INFO) parser = argparse.ArgumentParser(description='test synchronous write bandwidth') parser.add_argument('--block_size', '-b', default=16384*1024*10, type=int, help='block size') parser.add_argument('--iterations', '-i', default=100, type=int, help='iterations') args = parser.parse_args() data = os.urandom(args.block_size) logging.debug("opening file") f_out = os.open('test.dat', os.O_CREAT | os.O_WRONLY | os.O_SYNC) bytes_written = 0 elapsed = 0 logging.debug("starting test") start = time.time() for i in range(args.iterations): bytes_written += os.write(f_out, data) end = time.time() elapsed = end - start os.close(f_out) print("{0} bytes written in {1} seconds {2} MB/s".format(bytes_written, elapsed, 1.0E-6*bytes_written/elapsed)) logging.info("{0} bytes written in {1} seconds {2} MB/s".format(bytes_written, elapsed, 1.0E-6*bytes_written/elapsed)) |
Post a Comment