Are you recording acoustic or electric guitar? In case of electric will you mic up an amp or do you want to go directly into the computer? Are you also going to record vocals with the mic?
A good place to start would be a condenser microphone. I have an Audio Technica AT2020 which I use to record both acoustic guitar and vocals. If you plan on recording vocals make sure you also get a pop filter. Also it would be best to use it on a mic stand. There is a USB version of that mic, but be careful, because it limits you to only recording through the mic.
If, for example, you want to record electric guitar directly to your computer you're going to need an audio interface (I have an Edirol UA-25), either USB or firewire. There are lots of options here so look around and find something that fits you. What I would look for is an interface that has both XLR and regular instrument jack inputs and phantom power for condenser mics.
Check to see if there is any software bundled with the interface. For example: if you want to record electric guitar directly to the computer you could look at the Line6 UX1 or UX2 interfaces, which come with POD Farm software bundled, which gives you a full virtual guitar rig. I use POD Farm for all my guitar and bass sounds and am quite happy with it.
I am not familiar with Audacity, and how well it handles multitrack recording. I can recommend Reaper ( http://reaper.fm ) which is a fully featured digital audio workstation that is very lightweight (I used it without problems with XP before switching to Win7). You can download and use the evaluation version of Reaper without restrictions on features or time limit. But if you're going to use it regularly a license costs only $60.
Finally you should probably look at a decent set of earphones.
I hope this info was useful. I'm be no means an expert, but I have found a setup what works for me. If you have any questions let me know.
= new reply since forum marked as read
