By default Internet Explorer 11/10/9 in Windows 7 and Windows 8 / 8.1 shows a preview of the tabs opened when you mouse over the IE taskbar button, as shown on the screenshot. (This tutorial also applies to Windows 10, but it uses the Edge browser by default.) In my case, I disabled grouping taskbar buttons, so you see two buttons for two IE windows opened. By default, you'd see only one button, making difficult to switch between opened browser windows.
Depending on the total number of tabs you have opened, Internet Explorer may show a menu instead of preview thumbnails. The switch between the two is automatically done once you reach "x" number of web browser tabs. That number depends on your screen resolution: Internet Explorer will show a menu as soon as the taskbar thumbnail would need to become to small to fit on the screen's available width (whether or not you have multiple monitors).
While jump lists and taskbar preview thumbnails are part of Windows itself, Internet Explorer lets you disable showing individual tabs in the previews: in other words, once disabled, you would only see one thumbnail for each IE window opened, regardless of the number of tabs each one contains. Click on the Tools menu (gear icon in the top right corner of the window). You can hit the Alt+X keyboard shortcut, or Alt+T if you prefer to use IE's classic menus.
Select Internet options at the bottom, and click on the Tabs button when the Options dialog opens. The "General" tab should automatically be selected.
Under Tabbed Browsing at the top, uncheck the "Show previews for individual tabs in the taskbar" checkbox. The asterisk at the end means that the change won't takes effect until you restart Internet Explorer. That means that every single window must be closed before you relaunch it. You don't have to do it right now, but the taskbar thumbnail previews won't be hidden until you do! Click OK to save your tab settings, and OK once more to return to IE.
Bonus Tip: the "Enable tab groups" checkbox you see in the last checkbox is the setting to disable to prevent Internet Explorer from coloring tabs. By default, IE helps you know which tab opened which link by making them visually similar. I generally find this feature distracting, but it can be very useful when you open a website's links in background tabs, like reading news from multiple sources.