Agreed.  It serves no one except the flagger to remove rewards from a post.  The others voting for it, unless there's some kind of evidence for collusion to actually unjustly drain the rewards pool, are voting it because they value that post to the level they've voted.