Whether you manually added a contact or added a sender to your address book, Microsoft Outlook will store a digital business card for that person under Contacts. When you email someone for the first time (replying or composing a message to that person), Outlook will automatically add that email address to the suggested recipients list, available next time you type from the To, Cc, or Bcc fields (screenshot). I'll show you how to remove an entry for both cases.
Note: I'm using Outlook 2016 in this tutorial, but these steps work in Outlook 2013, Outlook 2010, and even earlier versions of the program! For a suggested email address, which isn't a contact, the solution is simple: type that person's name or email address in one of the recipient fields. Now mouse over that suggestion and hit the delete key (Del) when it is highlighted, or click on the tiny delete icon. As soon as you do, that suggestion is permanently removed.
Here's a trick: if you don't like that Outlook automatically adds as suggested contact anyone to whom you send a message, that can be turned off. From Outlook's main window, click on the File button (top left corner), and select Options. When the Options dialog opens, select "Mail" on the left. Under Send Messages, uncheck the "Use auto-complete list to suggest names when typing in the To, Cc, and Bcc lines" checkbox. Click OK to save your new setting.
Although you can select Contacts in the lower left corner of the window, or hit the equivalent keyboard shortcut of Ctrl+3, there's an even faster way. Press Ctrl+Shift+B: since Outlook 2003 (and maybe even earlier!), this opens the address book popup, from where you can perform a quick search. Once you found the contact, right-click on it and choose "Delete". The advantage of this approach is that you don't leave whatever you were doing in Outlook.
If you prefer to stay in Outlook's main window, hit Ctrl+3 or click on the Contacts icon, as shown on the screenshot. This loads a searchable list of all your contacts, sorted alphabetically. Just right-click on any of them and choose "Delete" to permanently remove them. Tip: to go back to your inbox, hit Ctrl+1 or click on the envelope icon (also visible in the screenshot). The Ctrl+2 hotkey goes to the Outlook calendar, and Ctrl+4 loads your tasks and flagged emails.