You may already know from previous tutorials that Internet Explorer's Favorites menu just shows the bookmark shortcuts contained in the Favorites folder under your Windows user account / profile. As such, removing bookmarks from your Favorites is as easy as deleting files from a folder. I'll show you three ways to delete saved website shortcuts you added to your Favorites in Internet Explorer 11/10/9 - two of these approaches also work in older versions of IE.
Delete bookmarked websites from the Favorites pane: in Internet Explorer 11/10/9, click on the star button / icon near the top right corner of the window, or hit the equivalent keyboard shortcut of Alt+C. The button shown in the screenshot lets you "Pin the Favorites Center" - in other words, dock that pane to the left side of the window. Right-click on any bookmark or favorite folder, and choose "Delete": it will move it (and its content) to the Recycle Bin.
Whether or not you are currently showing Internet Explorer's classic menus, they're always there: hit the Alt key to view them. The classic Favorites menu lets you reorder your Favorites, sort them alphabetically, and lets you delete them right from the menu itself! To delete a favorite or folder in the menu, just right-click on it and choose "Delete". Like before, this only moves them to the Recycle Bin, from where they can be restored, like any other file!
Finally, you can manage your IE bookmarks from Windows Explorer: click on the start button, and type "favorites" in the start menu's search field. Hit Enter when the system Favorites folder is visible at the top, or click on it. You can also type the word "favorites" and hit Enter in Windows Explorer's address bar (this also works with other folder names, like "pictures" or "desktop"!)
You are now in a plain old folder: just select the files you want to delete - each of them is a favorite (bookmark). Deleting a folder, predictably, will also delete all the shortcuts it contains. To rename a favorite or favorite folder, click on it and hit the F2 key, or right-click on it and choose "Rename". Reorganizing your IE favorites is much easier to do in Explorer, since you can easily create new folders, move them one into another, sort bookmarks by name / date, etc.