There are plenty of exchanges where you can buy and sell bitcoin as well as other cryptocurrencies. coinbase is one of my favorites simply because it is easy to use. Signing up and using coinbase is a simple process. After signing up, you have the option to connect to your bank account as well as a credit card. For security reasons, coinbase will require that you verify your accounts before you buy and sell. Visit coinbase, follow the signup instructions, and then wait until your account has been approved.
After you've been approved to buy bitcoin, log in to your coinbase account and click "Buy/Sell". You will see a page that lets you enter the amount of bitcoins you would like to purchase. On this same page you will see the USD conversion as well as any fees you will be charged if you decide to go through with the transaction. To continue with the transaction, select your payment method and your wallet. We will cover wallets in the future, but by default you can use the wallet that is attached to your coinbase account.
There is plenty of excellent bitcoin wallet software available. In fact, you can visit bitcoin.org to find links to a few of them. One program that is available on a wide range of systems is called Electrum. You can download here. Walk through the installation process and remember your seed as well as your password. I recommend that you generate a strong password and save it and the seed together in two locations.
Hopefully you realize by now that there are many possible ways to accomplish each of these steps. You can leave your bitcoin on the exchange of your choice. But I like to have more control so I transfer them to my wallet on my personal computer. If you're using Electrum you can simply open the application, and click on the tab labeled "Receive". You will find a number of addresses connected to your wallet. From your exchange or from any other wallet, you can send bitcoin to this address and you will see them arrive almost immediately. This transaction will have to be confirmed by the blockchain, but this doesn't usually take long. The address associated with this wallet can be used by anyone to send you bitcoin.
Most wallets give you the ability to backup or save a copy of your wallet. You should not consider this an optional step. Backup your wallet and save it twice. In this guide I'm not going to cover the how-to's of encryption, but I would recommend looking into encryption software like TrueCrypt or Bitlocker. After encrypting your backup, you can feel more secure about saving it on another device or even a shared storage service just in case your hard drive fails.