I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
I enjoy reading thoughtful content that I disagree with. I downvote based on perceived intent of the comment or post. If it’s just mean, hateful, trolling, wildly off-topic, or anything like that it will get a downvote.