That sounds like you're visualizing the lever being on your side of a pivot point, so when you push it down the other side goes up. Feels natural enough to me!
If the lever is on your side of the pivot point, you'd have to invert both horizontal and vertical axes. I don't have any data, but I certainly don't know anyone who plays with both axes inverted (in first person games).
I sometimes do if the camera is far enough back that I feel like I'm controlling it as it zooms around the character's head. Maybe that makes it not first person, idk.