1. Support. Is there really anybody there? Send them a question by email and see how long they take to respond. You want to know if they'll be there when you need them. Some offer 24-hour support and a few will respond quickly. Speedy customer service is rare and indicates a superior service. I think that this is the single most important consideration. Also, check out the quality of their online documentation.
2. Server performance. You want a server that is reliable and fast. There is a free tool available that allows you to evaluate this before making a decision! NetMechanic.com will measure the speed of access for all facets of access (network speed, DNS lookup, connect time, download time, and absence of timeouts). The "download time" statistic is the most important single item to compare. Collect your list of candidate hosting services and run side-by-side tests of all the candidates. Sample during peak and non-peak times for optimal comparison.
3. Are they familiar with the application you plan to use? If you're planning to use a particular application that requires special setup parameters (such as Online Merchant, a popular online store), make sure your potential host is familiar with the application. Otherwise you may find yourself spending a great deal of time trying to figure out how to configure it - perhaps never succeeding!
4. Traffic allowance. Compare your traffic allowances. Some services give you a small limit on the amount of material that can be downloaded from your site each month and charge you large amounts when that limit is exceeded. Some offer huge traffic allowances (several thousand MB/month, where 1,000 MB = 1 GB) for the same price as others that limit you to a few hundred MB/month. Sure you probably need less than a hundred MB/month now, but why put yourself in a position where you have to move when your traffic takes off? Plan ahead and allow for growth.
More...