Poojan's Tech Blog / Bonnie++ benchmarks on ZFS

Summary

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

server-with-zil

16G

187

18981

11319

515

193465

393.9

28805

+++++

+++

28823

29889

+++++

+++

31294

Latency

115ms

17445ms

8424ms

37225us

526ms

981ms

Latency

7910us

32078us

135us

23422us

66us

60us

server-without-zil

16G

183

17758

10223

524

197812

366.7

22379

+++++

+++

30282

17921

+++++

+++

26938

Latency

90085us

24368ms

5063ms

49834us

818ms

1802ms

Latency

151ms

175ms

172us

12008us

112us

28419us

Running on FreeBSD 9.0 with services samba syslogd crashplan ezjail powerd smartd disabled and 2XSSD ZIL/ZLOG

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server          16G   179  99  6704   1  4855   0   503  99 195540  11  61.1   3
Latency               119ms   18383ms   13060ms   21398us    1266ms     834ms
Version  1.96       ------Sequential Create------ --------Random Create--------
server              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29630  87 +++++ +++ 27077  97 21939  95 23913  99  9379  98
Latency             16551us     181ms     452us   18512us     159us     392us
1.96,1.96,server,1,1346540534,16G,,179,99,6704,1,4855,0,503,99,195540,11,61.1,3,16,,,,,29630,87,+++++,+++,27077,97,21939,95,23913,99,9379,98,119ms,18383ms,13060ms,21398us,1266ms,834ms,16551us,181ms,452us,18512us,159us,392us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

server 16G 179 99 6704 1 4855 0 503 99 195540 11 61.1 3

Latency 119ms 18383ms 13060ms 21398us 1266ms 834ms

Version 1.96 ------Sequential Create------ --------Random Create--------

server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 29630 87 +++++ +++ 27077 97 21939 95 23913 99 9379 98

Latency 16551us 181ms 452us 18512us 159us 392us

1.96,1.96,server,1,1346540534,16G,,179,99,6704,1,4855,0,503,99,195540,11,61.1,3,16,,,,,29630,87,+++++,+++,27077,97,21939,95,23913,99,9379,98,119ms,18383ms,13060ms,21398us,1266ms,834ms,16551us,181ms,452us,18512us,159us,392us

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

server

16G

179

6704

4855

503

195540

61.1

29630

+++++

+++

27077

21939

23913

9379

Latency

119ms

18383ms

13060ms

21398us

1266ms

834ms

Latency

16551us

181ms

452us

18512us

159us

392us

Disabled more services (cron):

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server          16G   187  99 18981   3 11319   3   515  99 193465  11 393.9   7
Latency               115ms   17445ms    8424ms   37225us     526ms     981ms
Version  1.96       ------Sequential Create------ --------Random Create--------
server              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 28805  86 +++++ +++ 28823  96 29889  94 +++++ +++ 31294  96
Latency              7910us   32078us     135us   23422us      66us      60us
1.96,1.96,server-with-zil,1,1346569394,16G,,187,99,18981,3,11319,3,515,99,193465,11,393.9,7,16,,,,,28805,86,+++++,+++,28823,96,29889,94,+++++,+++,31294,96,115ms,17445ms,8424ms,37225us,526ms,981ms,7910us,32078us,135us,23422us,66us,60us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

server 16G 187 99 18981 3 11319 3 515 99 193465 11 393.9 7

Latency 115ms 17445ms 8424ms 37225us 526ms 981ms

Version 1.96 ------Sequential Create------ --------Random Create--------

server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 28805 86 +++++ +++ 28823 96 29889 94 +++++ +++ 31294 96

Latency 7910us 32078us 135us 23422us 66us 60us

1.96,1.96,server-with-zil,1,1346569394,16G,,187,99,18981,3,11319,3,515,99,193465,11,393.9,7,16,,,,,28805,86,+++++,+++,28823,96,29889,94,+++++,+++,31294,96,115ms,17445ms,8424ms,37225us,526ms,981ms,7910us,32078us,135us,23422us,66us,60us

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

server

16G

187

18981

11319

515

193465

393.9

28805

+++++

+++

28823

29889

+++++

+++

31294

Latency

115ms

17445ms

8424ms

37225us

526ms

981ms

Latency

7910us

32078us

135us

23422us

66us

60us

Pool configuration:

  pool: tank
 state: ONLINE
 scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012
config:

	NAME                   STATE     READ WRITE CKSUM
	tank                   ONLINE       0     0     0
	  mirror-0             ONLINE       0     0     0
	    gpt/WD15EARS       ONLINE       0     0     0
	    gpt/WD20EARS       ONLINE       0     0     0
	  mirror-1             ONLINE       0     0     0
	    gpt/WD5000BPVT     ONLINE       0     0     0
	    gpt/Maxtor7Y250M0  ONLINE       0     0     0
	logs
	  gpt/tank_zil0        ONLINE       0     0     0
	  gpt/tank_zil1        ONLINE       0     0     0
	cache
	  gpt/tank_cache0      ONLINE       0     0     0

pool: tank

state: ONLINE

scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012

config:

NAME STATE READ WRITE CKSUM

tank ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

gpt/WD15EARS ONLINE 0 0 0

gpt/WD20EARS ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

gpt/WD5000BPVT ONLINE 0 0 0

gpt/Maxtor7Y250M0 ONLINE 0 0 0

logs

gpt/tank_zil0 ONLINE 0 0 0

gpt/tank_zil1 ONLINE 0 0 0

cache

gpt/tank_cache0 ONLINE 0 0 0

Without ZLOG (ZIL):

Pool configuration:

  pool: tank
 state: ONLINE
 scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012
config:

	NAME                   STATE     READ WRITE CKSUM
	tank                   ONLINE       0     0     0
	  mirror-0             ONLINE       0     0     0
	    gpt/WD15EARS       ONLINE       0     0     0
	    gpt/WD20EARS       ONLINE       0     0     0
	  mirror-1             ONLINE       0     0     0
	    gpt/WD5000BPVT     ONLINE       0     0     0
	    gpt/Maxtor7Y250M0  ONLINE       0     0     0
	cache
	  gpt/tank_cache0      ONLINE       0     0     0

pool: tank

state: ONLINE

scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012

config:

NAME STATE READ WRITE CKSUM

tank ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

gpt/WD15EARS ONLINE 0 0 0

gpt/WD20EARS ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

gpt/WD5000BPVT ONLINE 0 0 0

gpt/Maxtor7Y250M0 ONLINE 0 0 0

cache

gpt/tank_cache0 ONLINE 0 0 0

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server          16G   183  98 17758   3 10223   3   524  99 197812  11 366.7   7
Latency             90085us   24368ms    5063ms   49834us     818ms    1802ms
Version  1.96       ------Sequential Create------ --------Random Create--------
server              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22379  56 +++++ +++ 30282  92 17921  75 +++++ +++ 26938  86
Latency               151ms     175ms     172us   12008us     112us   28419us
1.96,1.96,server-without-zil,1,1346612666,16G,,183,98,17758,3,10223,3,524,99,197812,11,366.7,7,16,,,,,22379,56,+++++,+++,30282,92,17921,75,+++++,+++,26938,86,90085us,24368ms,5063ms,49834us,818ms,1802ms,151ms,175ms,172us,12008us,112us,28419us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

server 16G 183 98 17758 3 10223 3 524 99 197812 11 366.7 7

Latency 90085us 24368ms 5063ms 49834us 818ms 1802ms

Version 1.96 ------Sequential Create------ --------Random Create--------

server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 22379 56 +++++ +++ 30282 92 17921 75 +++++ +++ 26938 86

Latency 151ms 175ms 172us 12008us 112us 28419us

1.96,1.96,server-without-zil,1,1346612666,16G,,183,98,17758,3,10223,3,524,99,197812,11,366.7,7,16,,,,,22379,56,+++++,+++,30282,92,17921,75,+++++,+++,26938,86,90085us,24368ms,5063ms,49834us,818ms,1802ms,151ms,175ms,172us,12008us,112us,28419us

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

server

16G

183

17758

10223

524

197812

366.7

22379

+++++

+++

30282

17921

+++++

+++

26938

Latency

90085us

24368ms

5063ms

49834us

818ms

1802ms

Latency

151ms

175ms

172us

12008us

112us

28419us

Old Run (2012-09-01) with services enabled

Running on FreeBSD 9.0 with small (relatively slow) 2X 8GB SSD ZIL (log) & 64GB Cache. With services running.

server% bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server          16G   185  92  3293   3  4788   3   506  97 200079  18 752.3  23
Latency               311ms   35756ms      413s     117ms     434ms     568ms
Version  1.96       ------Sequential Create------ --------Random Create--------
server              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21912  86 +++++ +++ 32562  95 31781  92 +++++ +++ 31671  95
Latency             28470us     162us     546us   20424us      41us      78us
1.96,1.96,server,1,1346475285,16G,,185,92,3293,3,4788,3,506,97,200079,18,752.3,23,16,,,,,21912,86,+++++,+++,32562,95,31781,92,+++++,+++,31671,95,311ms,35756ms,413s,117ms,434ms,568ms,28470us,162us,546us,20424us,41us,78us

server% bonnie++

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

server 16G 185 92 3293 3 4788 3 506 97 200079 18 752.3 23

Latency 311ms 35756ms 413s 117ms 434ms 568ms

Version 1.96 ------Sequential Create------ --------Random Create--------

server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 21912 86 +++++ +++ 32562 95 31781 92 +++++ +++ 31671 95

Latency 28470us 162us 546us 20424us 41us 78us

1.96,1.96,server,1,1346475285,16G,,185,92,3293,3,4788,3,506,97,200079,18,752.3,23,16,,,,,21912,86,+++++,+++,32562,95,31781,92,+++++,+++,31671,95,311ms,35756ms,413s,117ms,434ms,568ms,28470us,162us,546us,20424us,41us,78us

 server% zpool status tank  pool: tank
 state: ONLINE
 scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012
config:

        NAME                   STATE     READ WRITE CKSUM
        tank                   ONLINE       0     0     0
          mirror-0             ONLINE       0     0     0
            gpt/WD15EARS       ONLINE       0     0     0
            gpt/WD20EARS       ONLINE       0     0     0
          mirror-1             ONLINE       0     0     0
            gpt/WD5000BPVT     ONLINE       0     0     0
            gpt/Maxtor7Y250M0  ONLINE       0     0     0
        logs
          gpt/tank_zil0        ONLINE       0     0     0
          gpt/tank_zil1        ONLINE       0     0     0
        cache
          gpt/tank_cache0      ONLINE       0     0     0

server% zpool status tank pool: tank

state: ONLINE

scan: resilvered 414M in 0h10m with 0 errors on Mon Aug 27 15:59:32 2012

config:

NAME STATE READ WRITE CKSUM

tank ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

gpt/WD15EARS ONLINE 0 0 0

gpt/WD20EARS ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

gpt/WD5000BPVT ONLINE 0 0 0

gpt/Maxtor7Y250M0 ONLINE 0 0 0

logs

gpt/tank_zil0 ONLINE 0 0 0

gpt/tank_zil1 ONLINE 0 0 0

cache

gpt/tank_cache0 ONLINE 0 0 0

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

server

16G

185

3293

4788

506

200079

752.3

21912

+++++

+++

32562

31781

+++++

+++

31671

Latency

311ms

35756ms

413s

117ms

434ms

568ms

Latency

28470us

162us

546us

20424us

41us

78us

Be the first to like.

Unlike

Poojan's Tech Blog

Bonnie++ benchmarks on ZFS

Summary

Running on FreeBSD 9.0 with services samba syslogd crashplan ezjail powerd smartd disabled and 2XSSD ZIL/ZLOG

Disabled more services (cron):

Without ZLOG (ZIL):

Old Run (2012-09-01) with services enabled

Post a Comment

Search

Meta

Blogroll

Top Liked Posts

Your IP Address

Categories