In case you run into this like I did
edit /boot/loader.conf and add: hw.pci.honor_msi_blacklist=0
If anyone is still looking to resolve this issue, it’s not an mrsas driver issue. It’s apparently a blacklist that still exists in FreeBSD to disable MSI-X when it detects it’s running in VMware due to some old limitation that used to exist in VMware. To find this work around I noticed the “mrsas0 MSI-x setup failed” in dmesg and how it would fall back to legacy interrupts. So I chased down MSI-x and VMware and came across the first link in bugs.freebsd.org. I believe this affects mrsas and all HBAs that use MSI-x.
I have confirmed this working on the following setup:
Perc H730 configured in HBA mode, cache disabled, passthrough via vmware
VMware vSphere 6.7
FreeNAS 11.2 U3 test VM with 10 vcpus and 16GB RAM.
To install FreeNAS first time either configure with 1 vcpu, then increase after installation and editing loader.conf, or detach the HBA, install FreeNAS and attach HBA after editing loader.conf post install.
Note: Changing /boot/loader.conf does not always work for some, however adding the variable to the system tunables did work.
For Optane 900P:
I found a simple fix for this issue by adding the Optane 900P device ID to passthru.map
– ssh to ESXi
– edit /etc/vmware/passthru.map
– add following lines at the end of the file:
# Intel Optane 900P&amp;lt;br&amp;gt;8086 2700 d3d0 false
– restart hypervisor
I can now pass through the 900P to Freenas 11.3 without issue
To over-provision the free space on the drives down to max ZIL size of 16 GB and assign write cache to this partition (Leaving free unallocated space on the drive to allow for higher write performance consistency and wear leveling by internal Intel firmware). This has the added benefit of using another 16 GB partition on the drive as another ZIL for another pool if you’d like, just understand, “What eggs you are putting in what basket” if a drive should fail. In the below example, each pool would get it’s own two partitions from the 900P’s in a mirrored config. But be warned, there is only so much bandwidth per device and all partitions need to share this bandwidth, so you may run into issues with write contention should more than one pool slam the devices with writes at a time.
# Make sure drives have all blocks unallocated, this is destructive!!! # Blocks on drive must be completely clear to allow wear leveling to see them as "free space" # Make sure you have the right device! PCIe devices may not enumerate in the order you expect gpart destroy -F nvd0 gpart destroy -F nvd1 gpart create -s GPT nvd0 gpart create -s GPT nvd1 gpart add -t freebsd-zfs -a 1m -l sloga0 -s 16G nvd0 gpart add -t freebsd-zfs -a 1m -l slogb0 -s 16G nvd1 # To read drive partition IDs glabel status | grep nvd0 #Output # gpt/sloga0 N/A nvd0p1 #gptid/b47f75d5-4d17-11ea-aa55-0050568b4d0b N/A nvd0p1 glabel status | grep nvd1 #Output # gpt/slogb0 N/A nvd1p1 #gptid/eff3f7f8-4d18-11ea-aa55-0050568b4d0b N/A nvd1p1 #Then use these IDs to add drives to pool (not currently available via 11.3 FreeNAS UI) zpool add tank log mirror gptid/b47f75d5-4d17-11ea-aa55-0050568b4d0b gptid/eff3f7f8-4d18-11ea-aa55-0050568b4d0b # If you had a second pool gpart add -t freebsd-zfs -a 1m -l sloga1 -s 16G nvd0 gpart add -t freebsd-zfs -a 1m -l slogb1 -s 16G nvd1 # Then continue to find drive ID's and add them to the second pool, etc... # Helpful commands gpart show -l nvd0 => 40 937703008 nvd0 GPT (447G) 40 2008 - free - (1.0M) 2048 33554432 1 sloga0 (16G) 33556480 904146568 - free - (431G) gpart show -l nvd2 => 40 937703008 nvd2 GPT (447G) 40 2008 - free - (1.0M) 2048 33554432 1 slogb0 (16G) 33556480 904146568 - free - (431G)