Skip to content

RAIDZ bonnie++ measurements

15-Oct-12

Just rebuilt my ZFS pool using RAIDZ:

So, time for another bonnie++ benchmark (all interfering service disabled including powerd):

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server 16G 177 98 47029 7 23311 7 493 99 283155 16 69.6 3 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 90277us 4091ms 2700ms 37197us 4020ms 515ms Latency 10644us 116us 111us 22075us 29us 66us
Be the first to like.

Bonnie++ benchmarks on ZFS

02-Sep-12

Summary

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server-with-zil 16G 187 99 18981 3 11319 3 515 99 193465 11 393.9 7 16 28805 86 +++++ +++ 28823 96 29889 94 +++++ +++ 31294 96
Latency 115ms 17445ms 8424ms 37225us 526ms 981ms Latency 7910us 32078us 135us 23422us 66us 60us
server-without-zil 16G 183 98 17758 3 10223 3 524 99 197812 11 366.7 7 16 22379 56 +++++ +++ 30282 92 17921 75 +++++ +++ 26938 86
Latency 90085us 24368ms 5063ms 49834us 818ms 1802ms Latency 151ms 175ms 172us 12008us 112us 28419us

Running on FreeBSD 9.0 with services samba syslogd crashplan ezjail powerd smartd disabled and 2XSSD ZIL/ZLOG

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server 16G 179 99 6704 1 4855 0 503 99 195540 11 61.1 3 16 29630 87 +++++ +++ 27077 97 21939 95 23913 99 9379 98
Latency 119ms 18383ms 13060ms 21398us 1266ms 834ms Latency 16551us 181ms 452us 18512us 159us 392us

Disabled more services (cron):

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server 16G 187 99 18981 3 11319 3 515 99 193465 11 393.9 7 16 28805 86 +++++ +++ 28823 96 29889 94 +++++ +++ 31294 96
Latency 115ms 17445ms 8424ms 37225us 526ms 981ms Latency 7910us 32078us 135us 23422us 66us 60us

Pool configuration:

Without ZLOG (ZIL):

Pool configuration:

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server 16G 183 98 17758 3 10223 3 524 99 197812 11 366.7 7 16 22379 56 +++++ +++ 30282 92 17921 75 +++++ +++ 26938 86
Latency 90085us 24368ms 5063ms 49834us 818ms 1802ms Latency 151ms 175ms 172us 12008us 112us 28419us

Old Run (2012-09-01) with services enabled

Running on FreeBSD 9.0 with small (relatively slow) 2X 8GB SSD ZIL (log) & 64GB Cache. With services running.

Version 1.96 Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
server 16G 185 92 3293 3 4788 3 506 97 200079 18 752.3 23 16 21912 86 +++++ +++ 32562 95 31781 92 +++++ +++ 31671 95
Latency 311ms 35756ms 413s 117ms 434ms 568ms Latency 28470us 162us 546us 20424us 41us 78us
Be the first to like.

Autohotkey for Exceed on Demand Alt keys

14-Jun-12

Exceed on Demand only sends the right Alt key to the viewed X desktop (Linux machine you’re logging into). I use this AutoHotkey script to send both Alt keys to Linux:


LAlt::RAlt
Be the first to like.

SSD investigations on my ZFS server

18-Feb-12

From a previous post, I noticed the SSD that I’m using in my server showed only a 50 MB/s write speed. That’s weird because the specs show around 145 MB/s.

So, I decided to investigate some more. I removed the SSD from the zpool and did some dd tests on it.


server# dd if=/dev/zero of=/dev/ada3 bs=4M count=32
32+0 records in
32+0 records out
134217728 bytes transferred in 1.587130 secs (84566307 bytes/sec)
server# dd if=/dev/zero of=/dev/ada3 bs=4M count=128
128+0 records in
128+0 records out
536870912 bytes transferred in 5.888121 secs (91178650 bytes/sec)
server# dd if=/dev/zero of=/dev/ada3 bs=4M count=1024
1024+0 records in
1024+0 records out
4294967296 bytes transferred in 45.720133 secs (93940394 bytes/sec)
server# dd if=/dev/zero of=/dev/ada3 bs=4M count=16384
^C6528+0 records in
6527+0 records out
27376222208 bytes transferred in 277.460367 secs (98667145 bytes/sec)

That last run (which I interrupted) showed 94 MB/s.

 

The disks I am using (in a mirror) as a ZIL are much smaller. Let’s check their write performance:

server# zpool status
pool: backup
state: ONLINE
scan: none requested
config:

NAME              STATE     READ WRITE CKSUM
backup            ONLINE       0     0     0
gpt/zfs-backup  ONLINE       0     0     0

errors: No known data errors

pool: tank
state: ONLINE
scan: scrub repaired 0 in 12h12m with 0 errors on Sat Feb 11 03:35:06 2012
config:

NAME                   STATE     READ WRITE CKSUM
tank                   ONLINE       0     0     0
mirror-0             ONLINE       0     0     0
gpt/WD20EARS       ONLINE       0     0     0
gpt/WD15EARS       ONLINE       0     0     0
mirror-2             ONLINE       0     0     0
gpt/WD5000BPVT     ONLINE       0     0     0
gpt/Maxtor7Y250M0  ONLINE       0     0     0
logs
mirror-1             ONLINE       0     0     0
gpt/zil0           ONLINE       0     0     0
gpt/zil1           ONLINE       0     0     0

errors: No known data errors
server# zpool remove tank mirror-1
server# dd if=/dev/zero of=/dev/gpt/zil0 bs=4M count=1024
1024+0 records in
1024+0 records out
4294967296 bytes transferred in 88.818498 secs (48356676 bytes/sec)

They are around 46 MB/s write speed. Hmm. If I were willing to give up the mirror on the ZIL, I could break up the 64 GB SSD into a two-part partition and use one part as a ZIL and the rest as the L2ARC cache.

I’d also free up a couple of SATA ports in the process–which is probably good, because I’m using a 4X SATA PCI card that’s probably very limited on bandwidth anyway. Rather than pushing this platform to its extreme, it might make sense to live with less.

Be the first to like.

FreeBSD 9.0 post-install Samba performance

21-Jan-12

Note that the Samba shares resides in a ZFS pool with dedup turned on. Since the blocks that make up the file being sent are probably already on the pool, it’s not necessarily writing the block data to disk. Using the same method as last time:

C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.

Took: 112.75 sec.

C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 198.55 sec.

Whoa. That’s way worse than before. Let’s re-run each:

C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.

Took: 62.88 sec.

C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 114.41 sec.

That’s better, but still not as good as before. One thing to note: I did not delete the file after the first run copied it over to the server. This shouldn’t matter much, since it’s in a snapshot of the zfs dataset somewhere, so the dedup algorithm should always return that the file doesn’t need to be saved to disk–although the algorithm itself probably takes quite a while to run on the server’s Atom D525 processor. Anyway, I deleted the file this time

C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 107.59 sec.

I’ve turned off dedup on the server:

server# zfs set dedup=off tank

I deleted the file on the server.


C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 58.01 sec.

Running again a little later with dedup back on. Deleted the file on the server:

C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 123.14 sec.

Deleted file again.

C:\Users\Poojan\Desktop>time_send
1 file(s) copied.

Took: 107.19 sec.

Note that the second run here is 16 seconds faster than the first. Seems like caching effects have a 16 second impact. The dedup on/off has a 9 second impact.

Check-summing on dedup makes a difference. I notice during the send transfer that with dedup on, kernel takes 100%  of a CPU (of 4 hyperthreads), whereas with it off, kernel only takes 3%. Since we have a multi-threaded CPU, Samba takes one thread and dedup (kernel) takes another. It doesn’t really impact things much (much being greater than 10%)–which is an assumption the ZFS designers made with dedup. The assumption being that CPU processing is more available than I/O bandwidth.

In my case, on my Atom D525 they may be about even. Note that the first time I ran the send, the time was 199 seconds. One hypotehsis is that in this case, the checksum had to run but the file still needed to be stored to disk–however, I’m almost certain that a snapshot of the data exists in the pool (I snapshot every 15 minutes). Another possibility is that there are some dedup data structures that zfs needs to create on the first write, and so the data structures were being created in addition to the checksums being run. I’m basically speculating here.

Be the first to like.

Subsonic can’t bind to address on FreeBSD dual stack

20-Jan-12

Either build Java without IPv6, or you need the following in /etc/rc.conf:

tomcat60_enable=”YES”
tomcat60_java_opts=”-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true”

(The java.awt.headless is if you’re running without an X console.)

Also, delete everyting in /usr/local/apache-tomcat-6.0/webapps/ except for subsonic and subsonic.war and rename these ROOT and ROOT.war respectively. Make sure www:www is the owner of everything (including your subsonic databases in /var/subsonic).

While you’re here, edit webapps/ROOT/WEB-INF/classes/ehcache.xml and change to (just an example):

<cache name=”musicFolderCache”
maxElementsInMemory=”128″
eternal=”true”
diskPersistent=”true”
overflowToDisk=”true”/>

<cache name=”chatCache”
maxElementsInMemory=”128″
eternal=”true”
diskPersistent=”true”
overflowToDisk=”true”/>

Be the first to like.

Corsair USB 3.0 16GB CMFVYGT3-16GB speed test

20-Jan-12

Looks basically like that other Corsair I bought a month ago. I basically bought another one so I can set up my FreeBSD root as a mirror of two 16 GB USB drives that are relatively fast for USB (even in a USB 2.0 port). This is in a USB 3.0 port:

Be the first to like.

Allow LAN-only password login to sshd

17-Jan-12

See here and then add this to /etc/sshsshd_config:

Match Address 192.168.1.0/24
PasswordAuthentication yes

This allows clients on the local network to login using password authentication while limiting the Internet in general to key-based authentication.

Be the first to like.

Samba FreeBSD 8.2 benchmarks (pre-upgrade to FreeBSD 9.0)

13-Jan-12

I’m about to upgrade to FreeBSD. While I csup the latest RELENG-9.0 branch, I’m looking at my Samba performance on 8.2.

I’m measuring a copy of FreeBSD-8.2-RELEASE-i386-dvd1.iso to the Samba server using the following batch file (taken from here):

@echo off

set starttime=%TIME%
set startcsec=%STARTTIME:~9,2%
set startsecs=%STARTTIME:~6,2%
set startmins=%STARTTIME:~3,2%
set starthour=%STARTTIME:~0,2%
set /a starttime=(%starthour%*60*60*100)+(%startmins%*60*100)+(%startsecs%*100)+
(%startcsec%)

:TimeThis
copy C:\Users\Poojan\Desktop\FreeBSD-8.2-RELEASE-i386-dvd1.iso.xz \\SERVER\Poojan\Downloads

set endtime=%time%
set endcsec=%endTIME:~9,2%
set endsecs=%endTIME:~6,2%
set endmins=%endTIME:~3,2%
set endhour=%endTIME:~0,2%
if %endhour% LSS %starthour% set /a endhour+=24
set /a endtime=(%endhour%*60*60*100)+(%endmins%*60*100)+(%endsecs%*100)+(%endcsec%)

set /a timetaken= ( %endtime% – %starttime% )
set /a timetakens= %timetaken% / 100
set timetaken=%timetakens%.%timetaken:~-2%

echo.
echo Took: %timetaken% sec.

C:\Users\Poojan\Desktop>time_copy.bat
1 file(s) copied.

Took: 77.34 sec.

That’s a 1751164811 byte file, so that’s 21.59 MiB/s. To receive:

@echo off

set starttime=%TIME%
set startcsec=%STARTTIME:~9,2%
set startsecs=%STARTTIME:~6,2%
set startmins=%STARTTIME:~3,2%
set starthour=%STARTTIME:~0,2%
set /a starttime=(%starthour%*60*60*100)+(%startmins%*60*100)+(%startsecs%*100)+
(%startcsec%)

:TimeThis
copy \\SERVER\Downloads\FreeBSD-9.0-RELEASE-amd64-dvd1\FreeBSD-9.0-RELEASE-amd64-dvd1.iso  C:\Users\Poojan\Desktop
set endtime=%time%
set endcsec=%endTIME:~9,2%
set endsecs=%endTIME:~6,2%
set endmins=%endTIME:~3,2%
set endhour=%endTIME:~0,2%
if %endhour% LSS %starthour% set /a endhour+=24
set /a endtime=(%endhour%*60*60*100)+(%endmins%*60*100)+(%endsecs%*100)+(%endcsec%)

set /a timetaken= ( %endtime% – %starttime% )
set /a timetakens= %timetaken% / 100
set timetaken=%timetakens%.%timetaken:~-2%

echo.
echo Took: %timetaken% sec.

C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.

Took: 109.23 sec.

This is a 2,388,531,200 byte file, so that’s 20.85 MiB/s.

I’ll repeat these later, when the csup is done–just in case the download on the server is hurting things.

OK: it’s done. Re-running:

C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.

Took: 52.72 sec.

C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.

Took: 43.23 sec.

So, that’s 43.2 MiB/s and 52.69 MiB/s. Trying it again, I get 42.09 MiB/s.

C:\Users\Poojan\Desktop>time_send.bat
1 file(s) copied.

Took: 73.38 sec.

C:\Users\Poojan\Desktop>time_send.bat
1 file(s) copied.

Took: 71.85 sec.

So, these are a bit more consistent: 22.76 MiB/s and 23.24 MiB/s.  Running it a third time gives 16.4 MiB, but I was running gstat, top, etc. while that was going.

2 people like this post.

Set up vimage/vnet jail on FreeBSD 8.2

08-Jan-12

With epair, there are two interfaces created epair0a and eapir0b that are direct connections between each other (like an Ethernet cable). When used with jails using an ifconfig vnet command, one side (epair0b for example) sgoes inside the jail. Since the other side of this virtual direct-connection stays on the outside of the jail, the epair pairs act as an Ethernet tunnel inside/outside the jail. You can use firewall rules either outside the jail or inside the jail to control traffic.

More…

4 people like this post.