I like how Homeworld does it. If you tell your group to attack a single enemy ship, the whole group attacks that ship. However, you can also tell it to attack a whole selection, in which case they'll spread out their firepower. Each of your vessels picks a target and then keeps attacking other members of the enemy group until they're all destroyed. I haven't played in a while, but if I remember correctly, you press the attack button and then hold alt while drawing a lasso around the target group.