Agreed. Everything checks out as far as I can tell, but I'm not that experienced at Blender's physics simulations.
@Gabbe: Just for kicks, try subdividing the trousers a few times and see if it fits more properly. I'd try it, but you've got the computer more suited for heavy physics calculations. If the poly count is kinda low, Blender will try to make the best approximation it can, and sometimes that involves ugly overlap. For example, you might drape a cloth over a table and the corners will stick straight through the cloth.