Correct me if I'm wrong, but would you need to edit the extractor code to extract your custom map contents? Since extracting maps only reads default MPQs and not custom patches. VMAPS requires maps and if you don't have the maps for your custom island it won't generate the vmaps.
I also have this issue. This is makes any creature you attack fall through the world.
My modified ad.exe displays "Unknown Compression" errors on my custom Map.dbc. Same with custom ADT files.