In case you run into this like I did

For H730:

Credit: https://www.ixsystems.com/community/threads/run_interrupt_driven_hooks-problem-with-mrsas-driver.61072/post-526236

edit /boot/loader.conf and add: hw.pci.honor_msi_blacklist=0

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203874https://redmine.ixsystems.com/issues/26733

If anyone is still looking to resolve this issue, it’s not an mrsas driver issue. It’s apparently a blacklist that still exists in FreeBSD to disable MSI-X when it detects it’s running in VMware due to some old limitation that used to exist in VMware. To find this work around I noticed the “mrsas0 MSI-x setup failed” in dmesg and how it would fall back to legacy interrupts. So I chased down MSI-x and VMware and came across the first link in bugs.freebsd.org. I believe this affects mrsas and all HBAs that use MSI-x.

I have confirmed this working on the following setup:
Dell R430
Perc H730 configured in HBA mode, cache disabled, passthrough via vmware
VMware vSphere 6.7
FreeNAS 11.2 U3 test VM with 10 vcpus and 16GB RAM.

To install FreeNAS first time either configure with 1 vcpu, then increase after installation and editing loader.conf, or detach the HBA, install FreeNAS and attach HBA after editing loader.conf post install.

Note: Changing /boot/loader.conf does not always work for some, however adding the variable to the system tunables did work.

For Optane 900P:

Credit: https://redmine.ixsystems.com/issues/26508#note-62

I found a simple fix for this issue by adding the Optane 900P device ID to passthru.map

– ssh to ESXi
– edit /etc/vmware/passthru.map
– add following lines at the end of the file:

# Intel Optane 900P<br>8086 2700 d3d0 false

– restart hypervisor

I can now pass through the 900P to Freenas 11.3 without issue

To over-provision the free space on the drives down to max ZIL size of 16 GB and assign write cache to this partition (Leaving free unallocated space on the drive to allow for higher write performance consistency and wear leveling by internal Intel firmware). This has the added benefit of using another 16 GB partition on the drive as another ZIL for another pool if you’d like, just understand, “What eggs you are putting in what basket” if a drive should fail. In the below example, each pool would get it’s own two partitions from the 900P’s in a mirrored config. But be warned, there is only so much bandwidth per device and all partitions need to share this bandwidth, so you may run into issues with write contention should more than one pool slam the devices with writes at a time.
Credit: https://www.ixsystems.com/community/threads/sharing-slog-devices-striped-ssds-partitioned-for-two-pools.62787/

PoolX
-nvd0p1
-nvd1p1
PoolY
-nvd0p2
-nvd1p2
etc…

# Make sure drives have all blocks unallocated, this is destructive!!!
# Blocks on drive must be completely clear to allow wear leveling to see them as "free space"
# Make sure you have the right device! PCIe devices may not enumerate in the order you expect

gpart destroy -F nvd0 
gpart destroy -F nvd1 

gpart create -s GPT nvd0
gpart create -s GPT nvd1

gpart add -t freebsd-zfs -a 1m -l sloga0 -s 16G nvd0
gpart add -t freebsd-zfs -a 1m -l slogb0 -s 16G nvd1

# To read drive partition IDs
glabel status | grep nvd0
#Output
#                                gpt/sloga0     N/A  nvd0p1
#gptid/b47f75d5-4d17-11ea-aa55-0050568b4d0b     N/A  nvd0p1

glabel status | grep nvd1
#Output
#                                gpt/slogb0     N/A  nvd1p1
#gptid/eff3f7f8-4d18-11ea-aa55-0050568b4d0b     N/A  nvd1p1

#Then use these IDs to add drives to pool (not currently available via 11.3 FreeNAS UI)
zpool add tank log mirror gptid/b47f75d5-4d17-11ea-aa55-0050568b4d0b gptid/eff3f7f8-4d18-11ea-aa55-0050568b4d0b 

# If you had a second pool
gpart add -t freebsd-zfs -a 1m -l sloga1 -s 16G nvd0
gpart add -t freebsd-zfs -a 1m -l slogb1 -s 16G nvd1

# Then continue to find drive ID's and add them to the second pool, etc...

# Helpful commands
gpart show -l nvd0
=>       40  937703008  nvd0  GPT  (447G)
         40       2008        - free -  (1.0M)
       2048   33554432     1  sloga0  (16G)
   33556480  904146568        - free -  (431G)

gpart show -l nvd2
=>       40  937703008  nvd2  GPT  (447G)
         40       2008        - free -  (1.0M)
       2048   33554432     1  slogb0  (16G)
   33556480  904146568        - free -  (431G)