Latency usually explodes as connection utilization goes up. If you have a router with good QOS and limit to 80% of your max available connection you're probably going to have pretty good latency.

Then again I'm not a networking guy. Someone who knows more about this stuff than I do can give you more comprehensive advice and talk about stuff like buffer bloat, etc.