By default, Photoshop shows black as foreground color, and white as background color. Unless you say otherwise, new PSD documents' background color is white. To create a transparent document, click on the "Background Contents" dropdown in the File > New dialog, and select "Transparent".
To change your swatch colors back to their default, hit D as keyboard shortcut on Windows or Mac OS X, and in any version of Photoshop for as long as I can remember! Or you can click the swatch icon, as shown in the screenshot.
To invert the two current colors (foreground color becomes background color, and vice versa), hit the X keyboard shortcut, or click on the swap colors icon - as I did in the screenshot. Note: the hotkey appears at the end of the tooltip.
To pick a different color, click on one of the two color squares to launch the color picker. Drag the sliders, switch color models, pick a color value. If you like that color, you can save it for later: click on the "Add to Swatches" button. Click OK. To show the color palette, which includes your custom color swatches, go to Window menu > Color. To adjust your color, just start over!
Here's a tip when you fill an entire canvas with color, or fill a selection with a color. You can hit Ctrl+Backspace -Windows- or Cmd+Delete (Mac) to fill the selection, or entire layer, with the background color (which would be blue, based on the screenshot). To fill the layer or selection with the foreground color, hit Alt+Backspace instead (Option+Delete on Mac). Or you can memorize one of these hotkeys, and just hit X to invert the colors when needed :)
Bonus Tips: you can use the Fill dialog to color the layer or selection with black or white even if they're not the current colors. Don't you hate having to temporarily switch to black and white just for one fill? Go to Edit > Fill, and pick any color (including custom), from the first dropdown. To use a color from your image, click on the toolbar swatch to open the Color dialog. Once opened, click to select the color you liked: your cursor will become an eyedropper tool!