Rural/Farm Life

If you can locate a very reputable trainer, it's well worth the money to have them work with you to find the right horse for your purposes. If you can, board with them and take lessons for a while as you learn not only how to ride the particular horse you acquire, but also learn as much as possible about proper horse care. Have a knowledgeable person help you find the right saddle, bit, etc., because something as simple as a tight saddle or the wrong bit may cause you a lot of heartache (or get you tossed).

As mentioned above, horses need good quality hay (scarcer and more expensive every day), a good pelleted feed, and often supplements. Shots and a yearly once-over run about $300 around here, plus you need to worm them regularly (actually now they suggest a vet check to target whatever variety they have at the moment), and there are always the unexpected very large bills if your horse develops a major sickness - colic, Potomac, EPM. This is a great site to learn about equine health issues:
http://www.thehorse.com/
You'll also need to have a good farrier stop by every six weeks or so to trim your horse's hooves.

We are down to one retired show horse, but still clean his stall and turn him out every day. Horses are herd animalsl and do best with others. Ours is quite lonely now, but it's good you plan on having goats. Horses & goats go well together.

Not trying to discourage you, but as mentioned by others above, you need to know what you're getting into. It's too easy to get the wrong horse or have your financial circumstances change, and then what do you do? Too many are being discarded because they just didn't work out ... or go hungry because owners underestimated the cost. Then they end up on Craigslist going to yet another bad situation, or going to a meat buyer for slaughter.

We're keeping our one because we love him too much to take the chance of letting him go to unknown circumstances, especially in this economy. So we have a 22 year old horse to care for for who knows how long - well into our retirement years (our other horse went strong till passing away at 33).

If you love horses and love to ride, I would just arm yourself with information and preferably a knowledgeable horseman before you jump in. The experience will be much more fun for you that way!