Whoa. That’s way worse than before. Let’s re-run each:

That’s better, but still not as good as before. One thing to note: I did not delete the file after the first run copied it over to the server. This shouldn’t matter much, since it’s in a snapshot of the zfs dataset somewhere, so the dedup algorithm should always return that the file doesn’t need to be saved to disk–although the algorithm itself probably takes quite a while to run on the server’s Atom D525 processor. Anyway, I deleted the file this time

I’ve turned off dedup on the server:

I deleted the file on the server.

Running again a little later with dedup back on. Deleted the file on the server:

Deleted file again.

Note that the second run here is 16 seconds faster than the first. Seems like caching effects have a 16 second impact. The dedup on/off has a 9 second impact.

Check-summing on dedup makes a difference. I notice during the send transfer that with dedup on, kernel takes 100%  of a CPU (of 4 hyperthreads), whereas with it off, kernel only takes 3%. Since we have a multi-threaded CPU, Samba takes one thread and dedup (kernel) takes another. It doesn’t really impact things much (much being greater than 10%)–which is an assumption the ZFS designers made with dedup. The assumption being that CPU processing is more available than I/O bandwidth.

In my case, on my Atom D525 they may be about even. Note that the first time I ran the send, the time was 199 seconds. One hypotehsis is that in this case, the checksum had to run but the file still needed to be stored to disk–however, I’m almost certain that a snapshot of the data exists in the pool (I snapshot every 15 minutes). Another possibility is that there are some dedup data structures that zfs needs to create on the first write, and so the data structures were being created in addition to the checksums being run. I’m basically speculating here.

(The java.awt.headless is if you’re running without an X console.)

Also, delete everyting in /usr/local/apache-tomcat-6.0/webapps/ except for subsonic and subsonic.war and rename these ROOT and ROOT.war respectively. Make sure www:www is the owner of everything (including your subsonic databases in /var/subsonic).

While you’re here, edit webapps/ROOT/WEB-INF/classes/ehcache.xml and change to (just an example):

-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    80.246 MB/s
          Sequential Write :    21.428 MB/s
         Random Read 512KB :    76.034 MB/s
        Random Write 512KB :     1.573 MB/s
    Random Read 4KB (QD=1) :    10.279 MB/s [  2509.5 IOPS]
   Random Write 4KB (QD=1) :     0.030 MB/s [     7.4 IOPS]
   Random Read 4KB (QD=32) :    10.862 MB/s [  2651.8 IOPS]
  Random Write 4KB (QD=32) :     0.032 MB/s [     7.8 IOPS]

  Test : 1000 MB [I: 0.0% (0.0/14.9 GB)] (x9)
  Date : 2012/01/19 22:51:41
    OS : Windows 7 Home Premium Edition SP1 [6.1 Build 7601] (x64)

This allows clients on the local network to login using password authentication while limiting the Internet in general to key-based authentication.

I’m measuring a copy of FreeBSD-8.2-RELEASE-i386-dvd1.iso to the Samba server using the following batch file (taken from here):

That’s a 1751164811 byte file, so that’s 21.59 MiB/s. To receive:

This is a 2,388,531,200 byte file, so that’s 20.85 MiB/s.

I’ll repeat these later, when the csup is done–just in case the download on the server is hurting things.

OK: it’s done. Re-running:

So, that’s 43.2 MiB/s and 52.69 MiB/s. Trying it again, I get 42.09 MiB/s.

So, these are a bit more consistent: 22.76 MiB/s and 23.24 MiB/s.  Running it a third time gives 16.4 MiB, but I was running gstat, top, etc. while that was going.

I’m using ezjail-admin with the following patch to /etc/rc.d/jail (taken from the FreeBSD Forums):

For example, I have two jails. I create the interfaces like so in /etc/rc.conf:

On the inside of the jail, I just need to set the IP address of the epair0b and epair1b interfaces. I'm using ezjail Here's a sample /usr/local/etc/ezjail/<jail_name>, in this case for a subsonic server:

Also, don't forget that you need something like the following within your jail's /etc/resolv.conf:

I'm using FreeBSD's IPFW. I have it set up so that it by default passes all packets. I can then control all firewall rules from the host (rather than in the jail). The last line of my host rules is a deny all to implementing the blocking on the host. I leave default rules within the jails (with epair/vnet each jail gets its own firewall rules). I have things set up like:

It's been a while since I set this up, and I'm not sure if the allow all on $oif (command 02000) is really necessary.

The configurations shown in the tables below have the formation [cache]_[gpart/gnop]_[zil]. Where [cache] is the L2ARC type. [gpart/gnop] says whether it’s set up under gpart, or direct. I call the direct method gnop since I have to initially use gnop to force a 4K sector size (I’m using the Western Digital Advanced Format drives for my pool). When I use gpart, I layer gnop on top of the initial creation to force 4K sector size. [zil] specifies the ZFS intent log device.

The disk types are:

Sandisk-2x8GB: these are two 8GB SATA SSD’s connected to the same controller (a PCI SATAII controller).
STT-2x8GB-usb: these are two USB 3.0 8GB drives connected to USB 2.0 ports.

I ran iozone using the command iozone -a -g 8G, and then looking only at the file sizes 1G or above. I have the whole results to make sure nothing funny happens for smaller file sizes. The results below are the average over all file sizes 1G or above and all record sizes performed at those file sizes.

I also computed the delta in bandwidth for a re-action versus action. For example, Delta Re-Write is throughput of re-write – throughput of write.

Despite some things I’ve read, USB 2.0 devices can adversely affect the performance. One can see that many cases with the USB devices used as caches, the re-read delta is 8MBps slower.

Another curiousity is that the ZIL devices actually slow things down for writes (and re-writes). I’m not sure if iozone uses synchronous writes, but I did see that the ZIL devices had activity during the tests, so I must conclude that something is going on there. Since I’m using this mostly as a Samba server, I’m probably going to skip the ZIL in my setup.

Here’s the controller/bus enumeration before any changes:

Curiously, I don’t think ahci1 (the Intel South Bridge) physically exposes all 4 channels on the motherboard. Only 2 SATA ports are available. Which doesn’t make sense, since Gigabyte explicitly put another 2-port SATA controller on the board

Originally, I had something like:

However, it seems like a “bus” as defined by hit.scbus isn’t the same as a SATA port. The system seems to enumerate all the SATA ports (from many controllers) on the same driver as ahcich0, ahcich1, etc.

Anyway, I rebooted and found that the above works

You’ll notice that scbus2 is unconnected. ada1 is on scbus1 and ada2 is on scbus3. That’s where the problem arises. If I were to connect another hard drive to scbus2, it would become the new ada2 and the previous ada2 would be shifted to ada3. Here’s how to assign them non-contiguous numbers:

Looks good. I get ada0, ada1, and skip ada2

Another thing I ran up against was that my 2 drives are not the same size. One is a 1.5TB and one is a 2TB. When this mess happened, the system put the 2TB in the place of the 1.5 TB (I think). This would have been fine, except, I couldn’t then re-add the 1.5TB drive as a replacement for the 2TB drive as it will only accept something the same size or bigger. This has been documented before here. For now, I’m throwing caution to the wind and not doing a gpart/glabel. Maybe I’ll regret it later, but I want to try this out (ZFS native only) for now. I’ve seen quite a bit on the forums that ZFS should handle everything natively.

USB 3.0 Corsair running in USB 3.0 (SuperSpeed)

Connected using a USB 3.0 port on the motherboard

Huh. Not bad for $25 (hopefully $15 after MIR)

Baseline SSD

Kingston Virtex SV100S264G SATA II 64GB SSD (Windows system drive)

USB 3.0 Corsair running in USB 2.0 (high speed)

Plugged into a USB 2.0 port on the same machine

For comparison SuperTech USB 3.0 8 GB in USB 3.0 (SuperSpeed)

I ran a USB 2.0 comparison on this drive before, except this time I’m running USB 3.0.

For comparison, SuperTech USB 3.0 8 GB in USB 2.0 (high speed)

Plugged into a USB 2.0 port on the same machine (replicates previous test).

The idea here is that I’m going to replace the ZFS cache on my server (which is presently a 64 GB SSD identical to the Kingston SSD benchmarked above) with the Corsair 16 GB USB 3.0. It will be running in USB 2.0, since my Atom D525 doesn’t have USB 3.0. However, I’d rather take the 64 GB SSD and stick it in a laptop, where the performance (and power) improvement will be more noticed. Right now, the L2ARC stats on the ZFS cache show around a 5% hit rate, so moving to a lower capacity cache probably won’t hurt too much. However, the slower read/write speeds probably will hurt a bit. We’ll see.

Here’s mine (tcsh):

Instead of casing screen, I should probably check for $DISPLAY. Someday.