{"id":131,"date":"2010-03-16T02:54:26","date_gmt":"2010-03-16T02:54:26","guid":{"rendered":"http:\/\/poojanwagh.opalstacked.com\/techblog\/un\/horrendous-freebsd-bonniee-latency\/"},"modified":"2010-03-16T11:33:07","modified_gmt":"2010-03-16T11:33:07","slug":"horrendous-freebsd-bonniee-latency","status":"publish","type":"post","link":"https:\/\/tech.poojanblog.com\/blog\/un\/horrendous-freebsd-bonniee-latency\/","title":{"rendered":"Horrendous FreeBSD bonniee++ latency"},"content":{"rendered":"

On my ZFS tank:
\n> bonnie++
\nWriting a byte at a time…done
\nWriting intelligently…done
\nRewriting…done
\nReading a byte at a time…done
\nReading intelligently…done
\nstart ’em…done…done…done…done…done…
\nCreate files in sequential order…done.
\nStat files in sequential order…done.
\nDelete files in sequential order…done.
\nCreate files in random order…done.
\nStat files in random order…done.
\nDelete files in random order…done.
\nVersion 1.96 ——Sequential Output—— –Sequential Input- –Random-
\nConcurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
\nMachine Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP \/sec %CP
\nfbsd2 2G 15 86 12165 15 7534 13 34 83 36594 27 76.8 10
\nLatency 7792ms 7856ms 9665ms 574ms 1311ms 106s
\nVersion 1.96 ——Sequential Create—— ——–Random Create——–
\nfbsd2 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 -Create– –Read— -Delete– -Create– –Read— -Delete–
\nfiles \/sec %CP \/sec %CP \/sec %CP \/sec %CP \/sec %CP \/sec %CP
\n16 3696 89 13108 92 3347 92 2572 80 5668 93 1679 93
\nLatency 129ms 11843us 5603us 573ms 8095us 7143us
\n1.96,1.96,fbsd2,1,1268703835,2G,,15,86,12165,15,7534,13,34,83,36594,27,76.8,10,16,,,,,3696,89,13108,92,3347,92,2572,80,5668,93,1679,93,7792ms,7856ms,9665ms,574ms,1311ms,106s,129ms,11843us,5603us,573ms,8095us,7143us
\n<\/code>
\nOn the SATA internal hard drive:
\n
\n> bonnie++
\nWriting a byte at a time…done
\nWriting intelligently…done
\nRewriting…done
\nReading a byte at a time…done
\nReading intelligently…done
\nstart ’em…done…done…done…done…done…
\nCreate files in sequential order…done.
\nStat files in sequential order…done.
\nDelete files in sequential order…done.
\nCreate files in random order…done.
\nStat files in random order…done.
\nDelete files in random order…done.
\nVersion 1.96 ——Sequential Output—— –Sequential Input- –Random-
\nConcurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
\nMachine Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP \/sec %CP
\nfbsd2 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0 2G 80 94 48456 34 15479 14 165 76 24437 12 119.8 7
\nLatency 114ms 363ms 351ms 119ms 138ms 68345ms
\nVersion 1.96 ——Sequential Create—— ——–Random Create——–
\nfbsd2 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0 -Create– –Read— -Delete– -Create– –Read— -Delete–
\nfiles \/sec %CP \/sec %CP \/sec %CP \/sec %CP \/sec %CP \/sec %CP
\n16 8027 70 27864 89 17448 71 6583 50 +++++ +++ 21427 86
\nLatency 183ms 31646us 13344us 320ms 274us 50409us
\n1.96,1.96,fbsd2,1,1268703414,2G,,80,94,48456,34,15479,14,165,76,24437 ,12,119.8,7,16,,,,,8027,70,27864,89,17448,71,6583,50,+++++,+++,21427,86,114ms,36 3ms,351ms,119ms,138ms,68345ms,183ms,31646us,13344us,320ms,274us,50409us
\n<\/code><\/p>\n

Thought maybe that automatic ZFS snapshots were getting in the way. But no:<\/p>\n
\nWriting a byte at a time…done
\nWriting intelligently…done
\nRewriting…done
\nReading a byte at a time…done
\nReading intelligently…done
\nstart ’em…done…done…done…done…done…
\nCreate files in sequential order…done.
\nStat files in sequential order…done.
\nDelete files in sequential order…done.
\nCreate files in random order…done.
\nStat files in random order…done.
\nDelete files in random order…done.
\nVersion\u00a0 1.96\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ——Sequential Output—— –Sequential Input- –Random-
\nConcurrency\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
\nMachine\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP\u00a0 \/sec %CP
\nfbsd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2G\u00a0\u00a0\u00a0 16\u00a0 85 12138\u00a0 14\u00a0 7793\u00a0 12\u00a0\u00a0\u00a0 39\u00a0 92 37740\u00a0 27\u00a0 78.2\u00a0 10
\nLatency\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 6945ms\u00a0\u00a0\u00a0 7693ms\u00a0\u00a0 21537ms\u00a0\u00a0\u00a0\u00a0 569ms\u00a0\u00a0\u00a0 1137ms\u00a0\u00a0\u00a0\u00a0\u00a0 104s
\nVersion\u00a0 1.96\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ——Sequential Create—— ——–Random Create——–
\nfbsd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -Create– –Read— -Delete– -Create– –Read— -Delete–
\nfiles\u00a0 \/sec %CP\u00a0 \/sec %CP\u00a0 \/sec %CP\u00a0 \/sec %CP\u00a0 \/sec %CP\u00a0 \/sec %CP
\n16\u00a0 4046\u00a0 83 14718\u00a0 93\u00a0 3581\u00a0 92\u00a0 3546\u00a0 91 12360\u00a0 94\u00a0 3029\u00a0 92
\nLatency\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 263ms\u00a0\u00a0\u00a0 6412us\u00a0\u00a0\u00a0 5649us\u00a0\u00a0\u00a0\u00a0 312ms\u00a0\u00a0\u00a0 5466us\u00a0\u00a0\u00a0 5879us
\n1.96,1.96,fbsd,1,1268710009,2G,,16,85,12138,14,7793,12,39,92,37740,27,78.2,10,16,,,,,4046,83,14718,93,3581,92,3546,91,12360,94,3029,92,6945ms,7693ms,21537ms,569ms,1137ms,104s,263ms,6412us,5649us,312ms,5466us,5879us
\n<\/code>\n

Test with bonnie<\/code>:<\/p>\nWriting with putc()…done
\nRewriting…done
\nWriting intelligently…done
\nReading with getc()…done
\nReading intelligently…done
\nSeeker 1…Seeker 2…Seeker 3…start ’em…done…done…done…<\/code>\n
<\/TH><\/TH>——-Sequential Output——–<\/TH>—Sequential Input–<\/TH>–Random–<\/TH><\/TR>
\n
<\/TH><\/TH>-Per Char-<\/TH>–Block—<\/TH>-Rewrite–<\/TH>-Per Char-<\/TH>–Block—<\/TH>–Seeks—<\/TH><\/TR>
\n		Machine<\/TH>MB<\/TH>K\/sec<\/TH>%CPU<\/TH>K\/sec<\/TH>%CPU<\/TH>K\/sec<\/TH>%CPU<\/TH>K\/sec<\/TH>%CPU<\/TH>K\/sec<\/TH>%CPU<\/TH>\/sec<\/TH>%CPU<\/TH><\/TR>
\n
fbsd<\/TD>8192<\/TD>8698<\/TD>34.7<\/TD>12017<\/TD>14.0<\/TD>8050<\/TD>12.9<\/TD>17828<\/TD>64.2<\/TD>38483<\/TD>25.7<\/TD> 54.5<\/TD> 1.6<\/TD><\/TR><\/table>\n<\/p>\n