By default, your iPhone / iPad / iPod touch will vibrate when you receive new messages in the iOS Mail app, in addition to the notification sound. I don't find the vibration helpful, and fortunately it can be turned off without affecting the sound notification. But this setting is definitely not where you'd expect it to be!
To disable new mail vibrate, launch the Settings app (gear icon). Navigate to its first screen, if needed, to see all the main options for your phone or tablet.
Scroll down until you see the Sounds settings, and tap on those.
Select "New Mail" on the next screen: these options let you configure both the sound notification that plays when you receive a new message (just tapping on any sound lets you preview it), but also lets you turn off the vibration. Apple could have stored it under the "Notifications" settings, but put it (only) under Sounds, for some reason, making hard to find unless you already know it!
iOS automatically scrolls down to the current sound for new mail. Scroll all the way back up, and tap on "Vibration - default" to customize that. You can pick a different vibration: tapping on one of them lets you preview it. To completely disable the new mail vibration, scroll all the way down and select "None". From now on, your iPhone won't vibrate any longer when you get new messages!