{"id":180,"date":"2010-04-04T04:11:31","date_gmt":"2010-04-04T04:11:31","guid":{"rendered":"http:\/\/poojanwagh.opalstacked.com\/techblog\/?p=180"},"modified":"2010-04-04T04:14:28","modified_gmt":"2010-04-04T04:14:28","slug":"adding-cache-device-to-zfs","status":"publish","type":"post","link":"https:\/\/tech.poojanblog.com\/blog\/unix-linux\/adding-cache-device-to-zfs\/","title":{"rendered":"Adding cache device to ZFS"},"content":{"rendered":"

I went ahead and bought this USB stick<\/a> to put in the back of my Portwell machine. The idea is to use it as a device cache to speed up my ZFS tank.
\nBased on this post by Alexander Leidinger<\/a>, it should reduce latency (when data is in the cache), but not otherwise. I was also curious on whether it would reduce power consumption. I have no idea if FreeBSD powers down USB devices when they are not being actively used.
\nI don’t write to the disks frequently. At most, there’ll be a read here and there. (I use Allway Sync which keeps some metadata on the remote storage, that does need to get read. However, like rsync, it won’t write unless necessary.)<\/p>\n

I had the Kill-a-Watt<\/a> running continuously (909 hours) before I did this switch, and it measured $29.92\/year (at 0.122 cents\/kWh, 25.95 kWh). When I looked at it, the Kill-a-Watt<\/a> registered 28 watts. (I think that\u2019s instantaneous, not average.)<\/p>\n

After doing so, I get the following dmesg<\/code>:<\/p>\nusbus0: 12Mbps Full Speed USB v1.0
\nusbus1: 12Mbps Full Speed USB v1.0
\nusbus2: 12Mbps Full Speed USB v1.0
\nusbus3: 480Mbps High Speed USB v2.0
\nad0: 152627MB <Hitachi HTE543216L9A300 FB2OC45C> at ata0-master UDMA33
\nugen0.1: <Intel> at usbus0
\nuhub0: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus0
\nugen1.1: <Intel> at usbus1
\nuhub1: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus1
\nugen2.1: <Intel> at usbus2
\nuhub2: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus2
\nugen3.1: <Intel> at usbus3
\nuhub3: <Intel EHCI root HUB, class 9\/0, rev 2.00\/1.00, addr 1> on usbus3
\nRoot mount waiting for: usbus3 usbus2 usbus1 usbus0
\nuhub0: 2 ports with 2 removable, self powered
\nuhub1: 2 ports with 2 removable, self powered
\nuhub2: 2 ports with 2 removable, self powered
\nRoot mount waiting for: usbus3
\nRoot mount waiting for: usbus3
\nRoot mount waiting for: usbus3
\nuhub3: 8 ports with 8 removable, self powered
\nugen3.2: <MICRONET> at usbus3
\numass0: <MICRONET FANTOM DRIVE, class 0\/0, rev 2.00\/2.10, addr 2> on usbus3
\numass0:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\nRoot mount waiting for: usbus3
\numass0:0:0:-1: Attached to scbus0
\nda0 at umass-sim0 bus 0 target 0 lun 0
\nda0: <MICRONET FANTOM DRIVE 2.10> Fixed Direct Access SCSI-4 device
\nda0: 40.000MB\/s transfers
\nda0: 476940MB (976773168 512 byte sectors: 255H 63S\/T 60801C)
\nugen3.3: <SMI Corporation> at usbus3
\numass1: <SMI Corporation USB, class 0\/0, rev 2.00\/0.00, addr 3> on usbus3
\numass1:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\numass1:1:1:-1: Attached to scbus1
\nda1 at umass-sim1 bus 1 target 0 lun 0
\nda1: <ST 16GB 0000> Removable Direct Access SCSI-0 device
\nda1: 40.000MB\/s transfers
\nda1: 15296MB (31326208 512 byte sectors: 255H 63S\/T 1949C)
\nusbd_set_config_index:523: could not read device status: USB_ERR_SHORT_XFER
\nugen3.4: <Simpletech> at usbus3
\numass2: <MSC Bulk-Only Transfer> on usbus3
\numass2:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\numass2:2:2:-1: Attached to scbus2
\nda2 at umass-sim2 bus 2 target 0 lun 0
\nda2: <Hitachi HDS721010CLA332 > Fixed Direct Access SCSI-2 device
\nda2: 40.000MB\/s transfers
\nda2: 953869MB (1953525168 512 byte sectors: 255H 63S\/T 121601C)
\nGEOM: da1: partition 1 does not start on a track boundary.
\nGEOM: da1: partition 1 does not end on a track boundary.
\nRoot mount waiting for: usbus3
\nugen3.5: <MICRONET> at usbus3
\numass3: <MICRONET FANTOM DRIVE, class 0\/0, rev 2.00\/2.10, addr 5> on usbus3
\numass3:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\nRoot mount waiting for: usbus3
\numass3:3:3:-1: Attached to scbus3
\nda3 at umass-sim3 bus 3 target 0 lun 0
\nda3: <MICRONET FANTOM DRIVE 2.10> Fixed Direct Access SCSI-4 device
\nda3: 40.000MB\/s transfers
\nda3: 953869MB (1953525168 512 byte sectors: 255H 63S\/T 121601C)
\nugen3.6: <vendor 0x13d1> at usbus3
\nugen2.2: <Cesko> at usbus2<\/code>\n

You\u2019ll notice from my previous post<\/a> that the drive that used to be da1 is now da2 and the drive that used to be da2 is now da3. zpool status shows:<\/p>\n> zpool status
\npool: tank
\nstate: UNAVAIL
\nscrub: none requested
\nconfig:<\/p>\n

NAME\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 STATE\u00a0\u00a0\u00a0\u00a0 READ WRITE CKSUM
\ntank\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 UNAVAIL\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0 insufficient replicas
\nraidz1\u00a0\u00a0\u00a0 UNAVAIL\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0 corrupted data
\nda0s1\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0
\nda1s1\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0
\nda2s1\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code>\n

I don\u2019t know for sure, but I think glabel<\/a> would have solved all this USB port-numbering nonsense<\/a>. The newly inserted USB key is now drive da1, bumping all the other drives up by one number. Anyway, at this point, it\u2019s fairly simple to play musical chairs (USB ports) with the last three drives. After doing so (and a fair amount of memory game with USB ports\/cables), I get the following dmesg<\/code>:<\/p>\nusbus0: 12Mbps Full Speed USB v1.0
\nusbus1: 12Mbps Full Speed USB v1.0
\nusbus2: 12Mbps Full Speed USB v1.0
\nusbus3: 480Mbps High Speed USB v2.0
\nad0: 152627MB <Hitachi HTE543216L9A300 FB2OC45C> at ata0-master UDMA33
\nugen0.1: <Intel> at usbus0
\nuhub0: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus0
\nugen1.1: <Intel> at usbus1
\nuhub1: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus1
\nugen2.1: <Intel> at usbus2
\nuhub2: <Intel UHCI root HUB, class 9\/0, rev 1.00\/1.00, addr 1> on usbus2
\nugen3.1: <Intel> at usbus3
\nuhub3: <Intel EHCI root HUB, class 9\/0, rev 2.00\/1.00, addr 1> on usbus3
\nRoot mount waiting for: usbus3 usbus2 usbus1 usbus0
\nuhub0: 2 ports with 2 removable, self powered
\nuhub1: 2 ports with 2 removable, self powered
\nuhub2: 2 ports with 2 removable, self powered
\nRoot mount waiting for: usbus3
\nRoot mount waiting for: usbus3
\nRoot mount waiting for: usbus3
\nuhub3: 8 ports with 8 removable, self powered
\nugen3.2: <MICRONET> at usbus3
\numass0: <MICRONET FANTOM DRIVE, class 0\/0, rev 2.00\/2.10, addr 2> on usbus3
\numass0:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\nRoot mount waiting for: usbus3
\numass0:0:0:-1: Attached to scbus0
\nda0 at umass-sim0 bus 0 target 0 lun 0
\nda0: <MICRONET FANTOM DRIVE 2.10> Fixed Direct Access SCSI-4 device
\nda0: 40.000MB\/s transfers
\nda0: 476940MB (976773168 512 byte sectors: 255H 63S\/T 60801C)
\nusbd_set_config_index:523: could not read device status: USB_ERR_SHORT_XFER
\nugen3.3: <Simpletech> at usbus3
\numass1: <MSC Bulk-Only Transfer> on usbus3
\numass1:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\numass1:1:1:-1: Attached to scbus1
\nda1 at umass-sim1 bus 1 target 0 lun 0
\nda1: <Hitachi HDS721010CLA332 > Fixed Direct Access SCSI-2 device
\nda1: 40.000MB\/s transfers
\nda1: 953869MB (1953525168 512 byte sectors: 255H 63S\/T 121601C)
\nugen3.4: <MICRONET> at usbus3
\numass2: <MICRONET FANTOM DRIVE, class 0\/0, rev 2.00\/2.10, addr 4> on usbus3
\numass2:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\numass2:2:2:-1: Attached to scbus2
\nda2 at umass-sim2 bus 2 target 0 lun 0
\nda2: <MICRONET FANTOM DRIVE 2.10> Fixed Direct Access SCSI-4 device
\nda2: 40.000MB\/s transfers
\nda2: 953869MB (1953525168 512 byte sectors: 255H 63S\/T 121601C)
\nugen3.5: <SMI Corporation> at usbus3
\numass3: <SMI Corporation USB, class 0\/0, rev 2.00\/0.00, addr 5> on usbus3
\numass3:\u00a0 SCSI over Bulk-Only; quirks = 0x0000
\numass3:3:3:-1: Attached to scbus3
\nda3 at umass-sim3 bus 3 target 0 lun 0
\nda3: <ST 16GB 0000> Removable Direct Access SCSI-0 device
\nda3: 40.000MB\/s transfers
\nda3: 15296MB (31326208 512 byte sectors: 255H 63S\/T 1949C)
\nugen3.6: <vendor 0x13d1> at usbus3
\nugen2.2: <Cesko> at usbus2
\nGEOM: da3: partition 1 does not start on a track boundary.
\nGEOM: da3: partition 1 does not end on a track boundary.
\nTrying to mount root from ufs:\/dev\/ad0s1a
\nWARNING: \/ was not properly dismounted
\nZFS filesystem version 13
\nZFS storage pool version 13<\/code>\n

Just to be safe, I then did a zpool scrub tank<\/code>. Okay, I got impatient and cancelled it after about 25% completion.<\/p>\n

After that, I used glabel to give the USB flash drive a persistent label (might as well start now): sudo glabel label -v pico_usb \/dev\/da3<\/code>. Then, I added the device as a cache:<\/p>\nsudo zpool add tank cache \/dev\/label\/pico_usb<\/code>\n

I\u2019m running a scrub now. Next, up, I\u2019ll use glabel on the other drives to have them have persistent labels (irrespective of which port I plug them into).<\/p>\n