I recently had a problem with ZFS. I went back to not using glabel, mainly because I wanted to force 4KB sector alignment on my drives and therefore used a gnop trick. About a month after doing so, I shuffled my drives around. I had ada4 and ada5 set up in a mirror configuration. At some point, I moved a drive around and ada4 became ada3 and ada3 became ada2. I could have (and should have) used the methods below to re-assign ada3/ada4 to their right positions. Anyway, I’m doing it now.
Here’s the controller/bus enumeration before any changes:
Dec 3 17:05:35 server kernel: ahci0: mem 0xfdefe000-0xfdefffff irq 17 at device 0.0 on pci2
Dec 3 17:05:35 server kernel: ahci0: [ITHREAD]
Dec 3 17:05:35 server kernel: ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
Dec 3 17:05:35 server kernel: ahcich0: at channel 0 on ahci0
Dec 3 17:05:35 server kernel: ahcich0: [ITHREAD]
Dec 3 17:05:35 server kernel: ahcich1: at channel 1 on ahci0
Dec 3 17:05:35 server kernel: ahcich1: [ITHREAD]
Dec 3 17:05:35 server kernel: atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 18 at device
Dec 3 17:05:35 server kernel: atapci0: [ITHREAD]
Dec 3 17:05:35 server kernel: ata2: on atapci0
Dec 3 17:05:35 server kernel: ata2: [ITHREAD]
...
Dec 3 17:05:35 server kernel: siis0: port 0xcf00-0xcf0f mem 0xfdcff000-0xfdcff07f,0xfdcf0000-0xfdcf7fff irq 20 at device 0.0 on pci3
Dec 3 17:05:35 server kernel: siis0: [ITHREAD]
Dec 3 17:05:35 server kernel: siisch0: at channel 0 on siis0
Dec 3 17:05:35 server kernel: siisch0: [ITHREAD]
Dec 3 17:05:35 server kernel: siisch1: at channel 1 on siis0
Dec 3 17:05:35 server kernel: siisch1: [ITHREAD]
Dec 3 17:05:35 server kernel: siisch2: at channel 2 on siis0
Dec 3 17:05:35 server kernel: siisch2: [ITHREAD]
Dec 3 17:05:35 server kernel: siisch3: at channel 3 on siis0
Dec 3 17:05:35 server kernel: siisch3: [ITHREAD]
...Dec 3 17:11:28 server kernel: ahci1: port 0xfa00-0xfa07,0xf900-0xf903,0xf800-0xf807,0xf700-0xf703,0xf600-0xf60f mem 0xfdffe000-0xfdff
Dec 3 17:11:28 server kernel: ahci1: [ITHREAD]
Dec 3 17:11:28 server kernel: ahci1: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
Dec 3 17:11:28 server kernel: ahcich2: at channel 0 on ahci1
Dec 3 17:11:28 server kernel: ahcich2: [ITHREAD]
Dec 3 17:11:28 server kernel: ahcich3: at channel 1 on ahci1
Dec 3 17:11:28 server kernel: ahcich3: [ITHREAD]
Dec 3 17:11:28 server kernel: ahcich4: at channel 2 on ahci1
Dec 3 17:11:28 server kernel: ahcich4: [ITHREAD]
Dec 3 17:11:28 server kernel: ahcich5: at channel 3 on ahci1
Dec 3 17:11:28 server kernel: ahcich5: [ITHREAD]
Curiously, I don’t think ahci1 (the Intel South Bridge) physically exposes all 4 channels on the motherboard. Only 2 SATA ports are available. Which doesn’t make sense, since Gigabyte explicitly put another 2-port SATA controller on the board
# Intel SATA controller (South Bridge)
hint.scbus.0.at="ahcich2"
hint.scbus.0.bus="0"
hint.scbus.1.at="ahcich3"
hint.scbus.1.bus="0"
# JMicron SATA controller (Gigabyte Motherboard)
hint.scbus.2.at="ahcich0"
hint.scbus.2.bus="0"
hint.scbus.3.at="ahcich1"
hint.scbus.3.bus="0"
# SiiS controller (PCI slot)
hint.scbus.4.at="siisch0"
hint.scbus.4.bus="0"
hint.scbus.5.at="siisch1"
hint.scbus.5.bus="0"
hint.scbus.6.at="siisch2"
hint.scbus.6.bus="0"
hint.scbus.7.at="siisch3"
hint.scbus.7.bus="0"
Originally, I had something like:
hint.scbus.0.at="ahci0"
hint.scbus.0.bus="0"
However, it seems like a “bus” as defined by hit.scbus isn’t the same as a SATA port. The system seems to enumerate all the SATA ports (from many controllers) on the same driver as ahcich0, ahcich1, etc.
Anyway, I rebooted and found that the above works
ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
ada0: ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich3 bus 0 scbus1 target 0 lun 0
ada1: ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich1 bus 0 scbus3 target 0 lun 0
ada2: ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada3 at siisch0 bus 0 scbus4 target 0 lun 0
ada3: ATA-7 SATA 1.x device
ada3: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
ada3: 7641MB (15649200 512 byte sectors: 16H 63S/T 15525C)
ada4 at siisch1 bus 0 scbus5 target 0 lun 0
ada4: ATA-7 SATA 1.x device
ada4: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
ada4: 7641MB (15649200 512 byte sectors: 16H 63S/T 15525C)
You’ll notice that scbus2 is unconnected. ada1 is on scbus1 and ada2 is on scbus3. That’s where the problem arises. If I were to connect another hard drive to scbus2, it would become the new ada2 and the previous ada2 would be shifted to ada3. Here’s how to assign them non-contiguous numbers:
hint.ada.0.at="scbus0"
hint.ada.0.target="0"
hint.ada.0.unit="0"
hint.ada.1.at="scbus1"
hint.ada.1.target="0"
hint.ada.1.unit="0"
hint.ada.2.at="scbus2"
hint.ada.2.target="0"
hint.ada.2.unit="0"
hint.ada.3.at="scbus3"
hint.ada.3.target="0"
hint.ada.3.unit="0"
hint.ada.4.at="scbus4"
hint.ada.4.target="0"
hint.ada.4.unit="0"
hint.ada.5.at="scbus5"
hint.ada.5.target="0"
hint.ada.5.unit="0"
hint.ada.6.at="scbus6"
hint.ada.6.target="0"
hint.ada.6.unit="0"
hint.ada.7.at="scbus7"
hint.ada.7.target="0"
hint.ada.7.unit="0"
Looks good. I get ada0, ada1, and skip ada2
Dec 17 10:24:10 server kernel: ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
Dec 17 10:24:10 server kernel: ada0: ATA-8 SATA 2.x device
Dec 17 10:24:10 server kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Dec 17 10:24:10 server kernel: ada0: Command Queueing enabled
Dec 17 10:24:10 server kernel: ada0: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
Dec 17 10:24:10 server kernel: ada1 at ahcich3 bus 0 scbus1 target 0 lun 0
Dec 17 10:24:10 server kernel: ada1: ATA-8 SATA 2.x device
Dec 17 10:24:10 server kernel: ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Dec 17 10:24:10 server kernel: ada1: Command Queueing enabled
Dec 17 10:24:10 server kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Dec 17 10:24:10 server kernel: ada3 at ahcich1 bus 0 scbus3 target 0 lun 0
Dec 17 10:24:10 server kernel: ada3: ATA-8 SATA 2.x device
Dec 17 10:24:10 server kernel: ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Dec 17 10:24:10 server kernel: ada3: Command Queueing enabled
Dec 17 10:24:10 server kernel: ada3: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
Dec 17 10:24:10 server kernel: ada4 at siisch0 bus 0 scbus4 target 0 lun 0
Dec 17 10:24:10 server kernel: ada4: ATA-7 SATA 1.x device
Dec 17 10:24:10 server kernel: ada4: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
Dec 17 10:24:10 server kernel: ada4: 7641MB (15649200 512 byte sectors: 16H 63S/T 15525C)
Dec 17 10:24:10 server kernel: ada5 at siisch1 bus 0 scbus5 target 0 lun 0
Dec 17 10:24:10 server kernel: ada5: ATA-7 SATA 1.x device
Dec 17 10:24:10 server kernel: ada5: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
Dec 17 10:24:10 server kernel: ada5: 7641MB (15649200 512 byte sectors: 16H 63S/T 15525C)
Another thing I ran up against was that my 2 drives are not the same size. One is a 1.5TB and one is a 2TB. When this mess happened, the system put the 2TB in the place of the 1.5 TB (I think). This would have been fine, except, I couldn’t then re-add the 1.5TB drive as a replacement for the 2TB drive as it will only accept something the same size or bigger. This has been documented before here. For now, I’m throwing caution to the wind and not doing a gpart/glabel. Maybe I’ll regret it later, but I want to try this out (ZFS native only) for now. I’ve seen quite a bit on the forums that ZFS should handle everything natively.
Post a Comment