Agreed. I use my downvote in cases of stalking, abuse, scams and the like. While I dislike people spamming their own content within my posts, I tend not to downvote them, but instead, write a warning, letting them know I will not read their posts if they spam them into comments. I think the whales misused their power and slammed innocent minnows who were struggling to get their posts seen. Giving people explanations instead of downvotes is a the proper thing to do in the cases of minnows seeking more exposure. Or at least if you downvote, let the person know why you're doing it, so they can learn for the future. This is basic emotional intelligence and proper use of empathic learning and teaching. Downvoting without explaning is a bad move and it is also insensitive. This only applies to people doing innocent self-promotion.
In the case of abusers, stalkers, and and scammers, of course, a heavy hand is necessary.