To the best of my understanding (and I don't claim to be an authority, I used to work in the field but there are folks that know orders of magnitudes more than I on the subject, and it's been a few years, and I'm easily confused and prone to wandering) there are several factors which can increase/decrease your overall latency. Among the important ones:
- your "last mile" ping time - how long it takes your modem to send/receive a packet of data to your ISP's server. This is where broadband (cable/DSL) connections shine compared to dial-up... sure they can download a lot more data per second, but as has been pointed out that isn't much of a factor here. But their ping times are a lot lower than a phone modem, and that is a Good Thing.
- the quality of your ISP's connection to the Internet backbone, in particular where and how it patches in
- the routing topography between there and the game server - the typical number and length of hops to get to the destination, and whether any of those hops are prone to packet loss (packet loss sucks a dead bear in summertime).