I'm hosting ez sites on my own server (would prefer having more than 1Gb, but sub sec results). I've tried before that to have hosted (hopeless) or virtual dedicated (virtually hopeless).
Nowing what runs on your server makes a huge difference indeed.