kasilag.me
Converting a Sun/Oracle F40/F80 Flash Accelerator to a LSI/Seagate Nytro Warpdrive

Why?

As a macOS user, I used to have a Fusion Drive setup, pairing a 128GB SATA SSD with a main 8TB storage disk (backed up of course with Time Machine to another 8TB disk).

But for some reason, APFS Fusion was really fussy about the kinds of drives it works with - data integrity went down the toilet after a single power failure.

So I wanted to move my boot drive to a "simpler" solution, which resulted in me finding the 800GB Sun F80 model. These things are going for cheap nowadays, at almost half the cost per GB compared to contemporary TLC(!) disks. It seems like these old enterprise "flash storage accelerators" are aging out of their lifespan in servers and getting unloaded on eBay by the bucket.

That's great, right?
Well, there's a small catch.

The Catch

LSI's Flash Memory division (now under Seagate) started selling these around 2012 as the Nytro Warpdrive, recycling their experience in producing SAS HBA's and RAID controllers... by sticking 4 SAS SSDs on an expansion card with a LSI SAS2008 controller running the show.

Ordinarily, this wouldn't be a problem since LSI's HBAs have some of the widest driver support available across OSes.
And did I mention that these are actually bootable from the BIOS, unlike some other early PCIe SSDs? (looks at FusionIO)

So what's the catch? Well, these cards weren't sold by LSI/Seagate. They're pulls from some sort of old Sun/Oracle server, and they're running custom firmware.
Instead of running in IR (integrated RAID) mode, they're running in IT (initiator target) mode, which presents each of the SSDs on board directly to the host OS. And for my use case, APFS RAID is an even more finicky and convoluted setup than APFS Fusion Drive.
There's just no way I was gonna run that configuration as a daily driver.

The Fix

Luckily, Sun/Oracle didn't change much else on these cards - they're perfectly compatible with the LSI/Seagate Nytro firmware.

User raileon on ServeTheHome posted about using IDA to force ddcli to flash the LSI/Seagate Nytro firmware on a F40 (400GB) card - but that apparently slowed the card down to PCIe 1x speed (~500MB/s).

Those speeds would kinda suck, and negate the whole point of getting a PCIe SSD in the first place, so I tried doing it the right way myself - ddcli actually reports that the controllers are compatible, but rejects flashing the card due to a PCI ID mismatch (108e:050a in the case of the Sun F80, compared to 1000:0504 in the case of the LSI/Seagate Nytro).
You can check the list of PCI IDs used in the series of Nytro cards yourself here.

Knowing that the only thing stopping ddcli were some pesky device ID checks, let's dive into actually doing the conversion:

Requirements

  • A Sun/Oracle F40/F80 flash accelerator
  • The latest Seagate Nytro firmware - get it for the F80, F40
  • A copy of Nytro Utility Management, which contains ddcli
  • A Linux installation with python3 and the build-essential metapackage
  • Compiled copies of lsirec (with sbrtool) and lsiutil
  • A decent hex editor.

Step One - Backup

YOU WILL LOSE ALL DATA BY SWITCHING FROM IT TO IR MODE, BACK UP ANY AND ALL DATA YOU HAVE ON THE CARD BEFORE STARTING!

If there's one thing I wish I did before jumping down this rabbit hole, it's doing more backups - the first time I did this I ended up totally bricking the SAS2008 with bad firmware, requiring me to use an E3 flasher (yes, the kind used on a PS3) to unbrick the onboard NOR flash.
You want to start by using lsiutil to back up the full 16MB NOR flash:

  • Run (as root):
    lsiutil -e
  • Select 46. Upload FLASH section then 5. Complete (all sections).
    Choose a filename, and keep the backups somewhere safe.
    You can also backup each section separately, if you want to.
  • Next, use # lspci -Dvvnn to get the full PCI device ID of your card. It should have "LSI Logic" somewhere in the device name.
  • With the full PCI ID in hand, we can now use lsirec to back up your card's SBR.
    Start by running # ./lsirec 0000:0X:00.0 readsbr sbr_backup.bin where 0000:0X:00.0 is your PCI device ID.
    Again, keep the backup somewhere safe.

Step Two - Prep Work

Now we're ready to do some real work. You need to extract the main controller firmware from the one you downloaded from Seagate - with other LSI SAS2008 based controllers the PCI option ROMs are shipped separately from the firmware; but on the Nytro it's shipped as a combined package.

  • Using a hex editor, open up the firmware you downloaded.
  • Copy the entire section starting from 000000EA 5AA5FA5A to 5AA5EA5A 4BFFF008.
    In my case, the NWD-BLP4-800 version 13.00.08.00 firmware has a SHA1 of 7f515610e0c51e47d2dc797b46145541152c75d7.
    Since the firmware is copyrighted material, I'm pretty sure you can't just post the pre-split binaries online :/ If anyone knows of a utility to split these combined firmware images, I'd really like to know about them!

Next, you need to edit your SBR to have the Nytro PCI IDs - this isn't strictly necessary for the conversion as the Nytro IR mode firmware will work just fine with the Sun PCI ID, but it does allow you to use the official ddcli utility to reflash the card.

  • Run $ python3 ./sbrtool.py parse sbr_backup.bin sbr.cfg
  • Edit sbr.cfg with your favorite text editor.
    You want to set PCIVID = 0x1000, PCIPID = 0x007e, SubsysVID = 0x1000 and SubsysPID = 0x0504 (800GB) or 0x0581 (400GB).
    Setting SASAddr was pointless in my experience, as it doesn't actually affect the SAS WWN that the card presents. Seems like that gets set in the card's VPD.
  • Next, build your new SBR by running $ python3 sbrtool.py build sbr.cfg sbr_new.bin

Step Three - Flashing

This is where it gets tricky - time to flash and do the actual conversion.

  • Run # ./lsiutil -e
  • Choose 33. Erase non-volatile adapter storage then 3. FLASH then 1. NVRAM and 8. Persistent manufacturing config pages
    Note that this also wipes out the VPD information containing SAS WWNs, you'll need to reprogram this region if you have more than 1 of these cards running.
  • Choose 2. Download firmware (update the FLASH) and select the extracted controller firmware you obtained in Step Two.
  • Exit lsiutil.
  • Run # ./lsirec 0000:0X:00.0 writesbr sbr_new.bin to flash the Nytro Warpdrive device IDs.
  • Reboot to Windows, or any of your favorite OSes that ddcli supports.
  • Get the card number of your device by running # ddcli -listall
  • Run # ddcli -c N -updatepkg NWD-BLP4-XXX_13.00.08.00.bin, where N is the number of your card, to properly install the full LSI/Seagate Nytro firmware, including the UEFI/BIOS option ROMs on your card.
  • Format your brand new Warpdrive by issuing # ddcli -c N -format to bring them into IR mode.
  • That's it! It should show up in your OS as one big, happy block device - running at the full PCIe x8 speeds it's capable of.

Credits