By default, Microsoft Outlook will show an envelope icon in Windows' system tray (notification area) when new email messages arrive. As a visual cue, it also opens the new mail popup, which can be disabled. On top of that, Outlook will play the new mail sound (you can also disable or change the new mail sound!) You can keep all notifications, some of them, or selectively disable them.
Windows lets you show or hide system tray icons, and that's a valid approach. To do that, click on the arrow button next to the system clock and click Customize. When the Notification Area Icons window opens, find "Microsoft Outlook" in the list (there are two of them: the second one controls the icon that shows when Outlook is hidden when minimized). Pick "Hide icon and notifications" in the dropdown next to the yellow envelope, and click OK.
Note: the envelope icon appears flat and yellow, because I'm using Outlook 2016 for this tutorial. The steps are the same for Outlook 2010 and Outlook 2013, but the envelope will look different. The hide-to-tray feature and new mail desktop alert have also remained the same for the last three versions of an already mature product. In fact, most changes were cosmetic since 2010!
But Microsoft Outlook also controls whether or not it shows an envelope through the program settings. From Outlook's main window, click on File (top left corner), and select Options in the left column of Backstage view.
Select the Mail settings on the left, and scroll down to the Message arrival options. Uncheck the Show an envelope icon in the taskbar checkbox and click OK. The yellow envelope will no longer appear in the system tray!