A million opinions on storage for ESXi over iSCSI from FreeNAS/TrueNAS, all boiled down

For those interested, and serving storage via FreeNAS/TrueNAS I’ve found a lot of bad guidance online about the zvol block size, serving iSCSI LUNs from a file in a dataset vs. a zvol, best underlying block (zvol)/record (dataset) size for VMs, etc.

What I’ve found to work best for virtual disks in VMware (YMMV but I’ve done a tremendious amount of research here):

• Use VMFS 6 with default 1 MB block size (VMFS will already shard sub-blocks)
• Set iSCSI.Max to 512
• Set IOPS to a value between 1 and 10 if using MPIO
  • Find your iSCSI device naa.xxxx
    • esxcli storage nmp device list
  • Set IOPS (The below is supposed to be one line)
    • for i in `esxcfg-scsidevs -c | awk '{print $1}' | grep naa.xxxx`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done
• Do not use Port Binding on iSCSI if your storage NICs are on separate subnets AND you have the physical nics from your ESXi hosts overridden to use only one uplink NIC or put iSCSI uplink NICs (each) on separate vSwitches
• Use Round Robin (RR) multipath policy
• Set your zvol block size (Cannot be changed unless zvol is destroyed and re-created) to 64k, this is really important as it will cause metadata thrashing in ZFS if too small or very high IOPS without much data transfered
• Leave your iSCSI Logical Block Size in FreeNAS/TrueNAS to 512 and leave unchecked “Disable Physical Block Size Reporting”
• LUN RPM in FreeNAS/TrueNAS should be almost always (even with spinners) set to SSD
• Once iSCSI device is mounted in ESXi, if SSD, don’t forget to manually mark the device as “Flash” from the host config

I have not played with Disk.DiskMaxIOSize (Default/Max: 32767 KB) or SCSI reservations yet.

https://www.suse.com/support/kb/doc/?id=000019221

https://vmdamentals.com/?p=2052