I know I'm going to suffer hard for this comment. However I'd suggest reading the ZFS optimisation guide.
I know the allure of raid 7 is strong, however if a disk were to fail on a LUM thats 24 disks wide, the chances of a rebuild is pretty low. (I know you've since changed that) the rebuild time is around 60-90 hours with no load.
(we use 14 disk raid6 LUNs for the balance of performance vs safety, however we've come pretty close to loosing it all.)
You'll get better performance if you have many smaller vdevs. You'll also compartmentalize the risk of multiple disk failure.
a raid 0+6 of four LUNs of 5 disks will have much greater performance, and will rebuild much faster, without risking too much. However you will loose more space
Nope, you're right on the money. My first thought was "this guy needs more VDEVs". Given the hardware and use case, 4x 6 disk RAIDZ2 VDEVs would make a ton more sense, and could also be cross-cabled to allow a controller failure.
I know the allure of raid 7 is strong, however if a disk were to fail on a LUM thats 24 disks wide, the chances of a rebuild is pretty low. (I know you've since changed that) the rebuild time is around 60-90 hours with no load.
(we use 14 disk raid6 LUNs for the balance of performance vs safety, however we've come pretty close to loosing it all.)
You'll get better performance if you have many smaller vdevs. You'll also compartmentalize the risk of multiple disk failure.
a raid 0+6 of four LUNs of 5 disks will have much greater performance, and will rebuild much faster, without risking too much. However you will loose more space