Poojan's Tech Blog

Note that the Samba shares resides in a ZFS pool with dedup turned on. Since the blocks that make up the file being sent are probably already on the pool, it’s not necessarily writing the block data to disk. Using the same method as last time:

Whoa. That’s way worse than before. Let’s re-run each:

That’s better, but still not as good as before. One thing to note: I did not delete the file after the first run copied it over to the server. This shouldn’t matter much, since it’s in a snapshot of the zfs dataset somewhere, so the dedup algorithm should always return that the file doesn’t need to be saved to disk–although the algorithm itself probably takes quite a while to run on the server’s Atom D525 processor. Anyway, I deleted the file this time

I’ve turned off dedup on the server:

I deleted the file on the server.

Running again a little later with dedup back on. Deleted the file on the server:

Deleted file again.

Note that the second run here is 16 seconds faster than the first. Seems like caching effects have a 16 second impact. The dedup on/off has a 9 second impact.

Check-summing on dedup makes a difference. I notice during the send transfer that with dedup on, kernel takes 100%  of a CPU (of 4 hyperthreads), whereas with it off, kernel only takes 3%. Since we have a multi-threaded CPU, Samba takes one thread and dedup (kernel) takes another. It doesn’t really impact things much (much being greater than 10%)–which is an assumption the ZFS designers made with dedup. The assumption being that CPU processing is more available than I/O bandwidth.

In my case, on my Atom D525 they may be about even. Note that the first time I ran the send, the time was 199 seconds. One hypotehsis is that in this case, the checksum had to run but the file still needed to be stored to disk–however, I’m almost certain that a snapshot of the data exists in the pool (I snapshot every 15 minutes). Another possibility is that there are some dedup data structures that zfs needs to create on the first write, and so the data structures were being created in addition to the checksums being run. I’m basically speculating here.

I’m about to upgrade to FreeBSD. While I csup the latest RELENG-9.0 branch, I’m looking at my Samba performance on 8.2.

I’m measuring a copy of FreeBSD-8.2-RELEASE-i386-dvd1.iso to the Samba server using the following batch file (taken from here):

That’s a 1751164811 byte file, so that’s 21.59 MiB/s. To receive:

This is a 2,388,531,200 byte file, so that’s 20.85 MiB/s.

I’ll repeat these later, when the csup is done–just in case the download on the server is hurting things.

OK: it’s done. Re-running:

So, that’s 43.2 MiB/s and 52.69 MiB/s. Trying it again, I get 42.09 MiB/s.

So, these are a bit more consistent: 22.76 MiB/s and 23.24 MiB/s.  Running it a third time gives 16.4 MiB, but I was running gstat, top, etc. while that was going.

I ran iozone on many different ZFS pool configurations, to get an idea of which drives are best for L2ARC (cache) and the ZIL. I also wanted to get an idea of whether using gpart affects performance.

The configurations shown in the tables below have the formation [cache]_[gpart/gnop]_[zil]. Where [cache] is the L2ARC type. [gpart/gnop] says whether it’s set up under gpart, or direct. I call the direct method gnop since I have to initially use gnop to force a 4K sector size (I’m using the Western Digital Advanced Format drives for my pool). When I use gpart, I layer gnop on top of the initial creation to force 4K sector size. [zil] specifies the ZFS intent log device.

The disk types are:

Sandisk-2x8GB: these are two 8GB SATA SSD’s connected to the same controller (a PCI SATAII controller).
STT-2x8GB-usb: these are two USB 3.0 8GB drives connected to USB 2.0 ports.

I ran iozone using the command iozone -a -g 8G, and then looking only at the file sizes 1G or above. I have the whole results to make sure nothing funny happens for smaller file sizes. The results below are the average over all file sizes 1G or above and all record sizes performed at those file sizes.

I also computed the delta in bandwidth for a re-action versus action. For example, Delta Re-Write is throughput of re-write – throughput of write.

Despite some things I’ve read, USB 2.0 devices can adversely affect the performance. One can see that many cases with the USB devices used as caches, the re-read delta is 8MBps slower.

Another curiousity is that the ZIL devices actually slow things down for writes (and re-writes). I’m not sure if iozone uses synchronous writes, but I did see that the ZIL devices had activity during the tests, so I must conclude that something is going on there. Since I’m using this mostly as a Samba server, I’m probably going to skip the ZIL in my setup.