Upvotes and downvotes are only relevant to the post and not the user now, since karma (thankfully) isn’t a thing. It even further encourages using downvotes as disagree buttons, since it doesn’t hurt the poster. But yeah, it also makes sense to downvote irrelevant posts instead. The problem is that there are only two buttons.