There are two types of Windows programs: some applications are "portable", or "standalone". These don't need to be installed: just unzip them to a folder and you are done. Removing these apps is just a matter of deleting their folder, just like deleting other files in Windows. But most programs require an uninstaller, because they create files all over the place (executable in Program Files folder, preferences in AppData folder, settings in the registry, etc.)
If you want to keep that program's preferences for possibly re-installing it later, see the tip at the very bottom (optional, and doesn't affect the uninstallation!)
To remove any program that was installed the standard way (with a step-by-step installer), open the start menu and type "uninstall" in the search field. Click on the "Uninstall a program" link result under Control Panel.
The Control Panel opens a list that shows all programs currently installed on your PC: click on the Name column on the left to sort them alphabetically, or "Installed On" to display most recently installed apps at the top. (Sorting by Size is useful if you're running out of space, but not too reliable.) You can also use the search box in the top right corner to quickly find by apps by name.
Right-click on the program you want to delete and choose "Uninstall" from the menu (nothing against the app pictured - I had to use an example :) Click Yes at the "Are you sure you want to uninstall [app name]" confirmation message, and enter the administrator's password to launch the uninstaller. What comes next depends on the app: just follow the prompts. By the end, the program is removed from the list, and its shortcuts deleted from the start menu.
Tip: often, removed programs don't fully clean up after themselves, even if the uninstaller explicitly offered to erase all its data and other settings. To make sure that your computer runs optimized (no dead entries in the registry, no temporary files piling up...), try running the Uninstaller Pro utility now and then - a free program that I use myself regularly. (Writing tutorials means that I keep installing and uninstalling new programs to test them out all the time!)
Technical: if you might re-install that program in the future, you can backup its settings (nearly all apps store information like recent files, and preferences like the font you want to use, etc.) Just open the start menu, type %appdata% in the search field, and hit Enter. When Windows Explorer opens, find that program's folder (it's usually named after the software or its maker). Copy that folder and paste it somewhere safe, so you'll be able to restore it.