Note that the Samba shares resides in a ZFS pool with dedup turned on. Since the blocks that make up the file being sent are probably already on the pool, it’s not necessarily writing the block data to disk. Using the same method as last time:
C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.
Took: 112.75 sec.
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 198.55 sec.
Whoa. That’s way worse than before. Let’s re-run each:
C:\Users\Poojan\Desktop>time_recv.bat
1 file(s) copied.
Took: 62.88 sec.
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 114.41 sec.
That’s better, but still not as good as before. One thing to note: I did not delete the file after the first run copied it over to the server. This shouldn’t matter much, since it’s in a snapshot of the zfs dataset somewhere, so the dedup algorithm should always return that the file doesn’t need to be saved to disk–although the algorithm itself probably takes quite a while to run on the server’s Atom D525 processor. Anyway, I deleted the file this time
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 107.59 sec.
I’ve turned off dedup on the server:
server# zfs set dedup=off tank
I deleted the file on the server.
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 58.01 sec.
Running again a little later with dedup back on. Deleted the file on the server:
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 123.14 sec.
Deleted file again.
C:\Users\Poojan\Desktop>time_send
1 file(s) copied.
Took: 107.19 sec.
Note that the second run here is 16 seconds faster than the first. Seems like caching effects have a 16 second impact. The dedup on/off has a 9 second impact.
Check-summing on dedup makes a difference. I notice during the send transfer that with dedup on, kernel takes 100% of a CPU (of 4 hyperthreads), whereas with it off, kernel only takes 3%. Since we have a multi-threaded CPU, Samba takes one thread and dedup (kernel) takes another. It doesn’t really impact things much (much being greater than 10%)–which is an assumption the ZFS designers made with dedup. The assumption being that CPU processing is more available than I/O bandwidth.
In my case, on my Atom D525 they may be about even. Note that the first time I ran the send, the time was 199 seconds. One hypotehsis is that in this case, the checksum had to run but the file still needed to be stored to disk–however, I’m almost certain that a snapshot of the data exists in the pool (I snapshot every 15 minutes). Another possibility is that there are some dedup data structures that zfs needs to create on the first write, and so the data structures were being created in addition to the checksums being run. I’m basically speculating here.
Post a Comment