KIP-85: Add an Automatic Review Layer With Loss-Only (Jurors) Reviewers

Proposal
1

Status: Draft:

  1. Summary:

This proposal adds an automatic review step after every jury vote in Kleros. A small set of reviewers is drawn to vote agree or disagree with the ruling. Reviewers can only lose or keep their stake. They never gain anything from reviewing.

If majority of the reviewers agree with the ruling, the decision becomes final for that court level. If majority of the reviewers disagree with the jury, the dispute proceeds to a regular Kleros appeal with a new jury. Reviewers from the review round cannot serve in the appeal or subsequent juries for the dispute. This process repeats at higher courts until either a review round majority agrees with the triggering jury ruling or the General Court is reached.

The goal is to strengthen the appeal system and reduce user appeal costs by shifting correctness incentives onto reviewers and jurors.

  1. Motivation

The current appeal system requires parties to manually watch, initiate, and pay for appeals. Incorrect rulings may go unchallenged because the cost barrier is high or due to obscurity. However, attempts to automate appeals create biased incentives for reviewers if they can profit from overturning. (You can think about this yourself to see why.)

A loss-only review layer solves this. Reviewers participate knowing they cannot profit from reviewing. Their only incentive is to avoid being wrong. This provides automatic quality control without distorting incentives.

  1. Specification

a. After each ruling, a review round is automatically opened. Reviewers vote either agree or disagree with the ruling.

b. Reviewer incentives:
Correct review vote: retain stake
Incorrect review vote: lose stake
No rewards of any kind are given to reviewers.

c. If the majority of reviewers agree with the ruling, the dispute finalizes at that level. Any dissenting reviewer’s stake goes to the triggering jury (increasing reward for the voting jury)

d. If the majority disagree, the dispute enters a regular appeal round in the same or higher court as defined by existing Kleros rules. The reviewers from the review round are excluded from serving in the appeal jury.

e. After the appeal jury rules, another review round occurs automatically. The same logic applies.

f. The process repeats up the court hierarchy until a review round agrees with the current ruling or the General Court rules and completes its review.

g. The number of reviewers work just like the current appeal process. I.E. When one juror votes, the reviewer count is 3 reviewers, etc.

  1. Rationale

This design avoids the incentive problems that arise when reviewers can earn rewards. There is no incentive to overturn for profit and no incentive to affirm for consistency rewards. The only goal of reviewers is not to be wrong, and, to vote, since mistakes cost stake.

The review layer provides fast, automated quality checks. Appeals only occur when the majority of reviewers disagree, reducing unnecessary escalation. The system remains compatible with existing Kleros appeal mechanics.

  1. Implementation Notes

Requires adding a review phase after voting, selecting reviewers (jurors), applying slashing to incorrect reviewers, excluding reviewers from the next appeal jury, and looping the process through higher courts.

  1. Governance

Requires standard Snapshot approval, Futarchy evaluation, development implementation, and a Governor upgrade.

  1. Backward Compatibility

Existing disputes continue under current rules. New disputes in configured courts follow the new ruling-review-appeal flow.

  1. Benefits

Improves ruling accuracy, adds a built-in appeal process that STILL allows for outside parties to initiate appeal while improving honest voting juror incentives.

Voting Options

Yes
No

2

why would i participate in becoming a reviewer if it just cost me money and time? i think lots of users would opt out especially if the cases are complicated.

what about improving the incentive structure for disputes and making crowdfunding disputes easier?

3

If you opted out with that logic, it would still help the overall network regardless.

Essentially, we don’t want reviewers or jurors who are thinking short-term.

Additionally, we are increasing the incentives for the HONEST voting jurors.

In other words, you would stay in the network for the rewards from being a Juror, just like it currently works. And note, this change actually increases the awards for voting jurors, at the expense of reveiwing jurors. If the reviewers vote wrong, you get their PNK.

So, the only way you “lose” is if you’re only ever a reveiwer. However, all Jurors are BY DEFAULT Jurors and Reviewers, and therefore there is no real loss to any of them.

This change should strongly bolster the Kleros Court, increase the reward for accurate rulings, and therefore lead to a price appreciation for all involved, due to the stronger wholistic ecosystem.

I’d like if @William reviewed this idea, and considered it with his research.

4

I think this is at least mostly compatible with the v2 contracts that are currently undergoing audit. Namely, that you could make a dispute kit that has this kind of appeal logic. From the perspective of the KlerosCore contract (which handles very broad functions related to the structure of the court tree and manages jurors’ staking transactions) all of this would be taking place during the appeal period.

We have generally not wanted to give small panels of jurors in early rounds the power to make a decision definitive and non-appealable, even if, for example, their ruling is unanimous. In that case an attacker would have a relatively small number of people they would need to bribe to win the case; you also have some chance that you get weird rulings because a small enough randomly drawn panel is just not representative of the community. Blocking further appeals if the review panel agrees with the ruling has similar issues, but those are mostly problems for high-value cases. For lower-value cases (the types of cases where incorrect rulings go unchallenged due to obscurity), something like this could make sense.

Another relevant point, we have thought a lot about how to generalize Kleros mechanisms to make them better at handling non-binary disputes. If we applied the sort of voting and incentive rules we have been developing for those use cases to the review panel, that could lead to issues as the ruling and the review may arrive at answers that are technically different but not fundamentally in disagreement. However, you could only give the review panel the mandate to answer a question like “did the voting round follow acceptable practices and is their ruling reasonable?” That would be a more procedural rule similar to questions that appeals courts focus on in some common law jurisdictions. That would binarize the question for the reviewers, so that could be complementary to the mechanisms the voting round uses.

So, I could definitely see this for some use cases. An important challenge is that, if the review triggers an appeal, there is still the question of who pays for the fees that the jurors in the appeal round receive. In the current mechanisms, fees are always paid on a per-case, per-round level. So each case is guaranteed to cover what it needs to compensate its jurors, and there is no large amount of ETH accumulating on the contracts that could be a honeypot for attacks. If you want to be able to trigger an appeal without putting in new fees, the court contract would need to have a reserve to pay that out of. So that would mean that contract taking some percentage fee on cases there were no appeals to be able to cover the cases where there are. That should be possible, but making sure it remains balanced requires a certain amount of care.

1 Like