While screenshots for this tutorial were taken with Outlook 2016, the exact same steps work with Outlook 2010 and Outlook 2013. By default, so as not to confuse users, Microsoft doesn't make Outlook automatically minimize itself to the tray. In either case, you'll still get by default the new mail popup notification, whenever new messages are downloaded from the server, making it quick and easy to restore Outlook and/or open a message whenever you need!
Let me first quickly show you how to make sure that Outlook's icon is always visible next to the clock. It's important, because if Outlook is minimized to the tray, clicking its icon is the only way you can restore Outlook (short of manually re-launching it from the start menu!) By the way, you can make Windows always show or hide any icons in the system tray, not just Outlook's.
Click on the arrow button, like I did in the first screenshot, and click Customize. Find the "Microsoft Outlook" entry in the listing that shows Outlook's logo as icon, and pick "Show icon and notifications" from its dropdown. You can also do the same for the new mail envelope icon (see screenshot), to guarantee that you'll always see when new messages arrive. And by default, Outlook will also display the new mail popup, and play the new mail sound.
Click OK to save. Now right-click on Outlook's icon and pick "Hide When Minimized" from the menu. If that was checked before you clicked, Outlook is already set to minimize itself to the tray. Go to Outlook and minimize it. The first time you do after enabling the "minimize to tray" feature, you'll get a bubble notification telling you that you can restore it by double-clicking on its icon.
Since you probably leave Outlook open most of the day, this is a great option, which saves you some space in the taskbar for other programs! As a bonus tip, know that you can also restore hidden Outlook by clicking or right-clicking once on its icon: when the context menu opens, choose "Open Outlook". Tip: you can make Outlook always start minimized. With the minimize-to-tray option enabled, this means that Outlook will basically start hidden in the tray!