After some hours of hacking I now have fully working ESXi 5.1 virtual machine setup with DS 4.2 (3202).
To get PVSCSI and VMXNET3 working a new kernel needs to be compiled. To make matters worse, scsi drives are not supported at all. So some steps need to be done:
scsi disk support must be added to drivers/scsi/sd.c so that they are marked as SYNO_DISK_SATA instead of SYNO_DISK_UNKNOWN
disks must be marked as system disks so that the md will autodetect the system raid partition (gd->systemDisk = 1)
having ata as well as scsi disks will not work. quick workaround is to disable sata support from the kernel. this has the added benefit of hiding the boot disk partitions
if sata is disabled some important symbols are not compiled in and synobios.ko module will not load, causing scemd to halt the system
implement dummy synobios kernel module that keeps scemd happy
Performance seems to be excellent, quite possibly due to paravirtualized drivers. S.M.A.R.T. is not working, but that was expected.
Some things are still in todo-list:
compile in open-vm-tools to keep esxi from nagging about installing tools
clean up kernel from unneeded device drivers, also clean modules from hd image
try to get scemd not to constantly poll SYNOIO_SET_DISK_LED and SYNOIO_GET_EUNIT_TYPE
investigate if using stock kernel is possible with dummy synobios and CONFIG_EXT4_FS_SYNO_ACL