As far as rankings go, you have to remember that Google's number one goal is to give their users a great experience. So, if your pages are returning errors, it will definitely affect your rankings. The last thing that Google wants to do is take its users to a broken page.
That being said, it does take quite some time for Google to crawl a page and notice that it is broken, and they do seem to give a bit of grace as sometimes pages are down for a short time. But if it stays down, it can completely wipe out the rankings of that particular page.