How about leaving down and up vote, but including a button for "disagree", which doesn't affect karma. I would use "disagree" instead of downvoting quite often. Leave the downvote button with its karma loss. That lets people downvote trolls/spams/irrationality, but lets the person being disagreed with that something's wrong. A "disagree" button may also help people to write a reply with the reason for disagreeing, rather than just downvoting and moving on, which in my experience happens a lot.