Predicting Outcomes of Case-based Legal Arguments

1 Predicting Outcomes of Case-based Legal ArgumentsStefanie Br uninghaus and Kevin D. AshleyLearning Research and Development Center,Intelligent Systems Program, School of LawUniversity of Pittsburgh, Pittsburgh, PA this paper, we introduce IBP, an algorithm that combinesreasoning with an abstract domain model and case -basedreasoning techniques to predict the outcome of case -basedlegal Arguments . Unlike the predictions generated by statis-tical or machine-learning techniques, IBP s predictions areaccompanied by describe an empirical evaluation of IBP, in which wecompare our algorithm to prediction based on Hypo s andCATO s relevance criteria, and to a number of widely usedmachine learning algorithms. IBP reaches higher accuracythan all competitors, and hypothesis testing shows that theobserved differences are statistically significant.

2 An abla-tion study indicates that both sources of knowledge in IBPcontribute to the accuracy of its Predicting case OUTCOMESMost AI and Law models of Case-based Legal reasoninghave eschewed Predicting the results of new cases in favorof making Arguments , , HYPO (Ashley 1990), CATO(Aleven 1997), GREBE (Branting 1999), CABARET (Riss-land & Skalak 1989); but see (Aleven 2003) and (Popple1993). Early projects (McKaay & Robillard 1974) used lo-gistic regression or basic nearest-neighbor methods for pre-diction, but did not employ Legal reasoning focus of Legal reasoning research on argumentationis understandable. The goals of automatically generatingarguments about a new case and Predicting its outcome co-exist in uneasy, although potentially productive, argumentation is fundamentally normative.

3 The goalof Case-based AI models of Legal reasoning is to generatereasonable precedential Arguments and counterarguments,regardless of empirical likelihood. Even when an advocatecites a past similar case as an authority, the assumptionsare that the court got it right in the precedent and thatthe advocate can show how the normative reasoning mapsonto the new case (Hafner & Berman 2002; Ashley 2002) oreven determines its outcome (Bench-Capon & Sartor 2001;Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a 200X ACM X-XXXXX-XX-X/XX/XX.)

4 $ 1999). The point of an advocate s distinguishing thecited case is to show factual differences that warrant a dif-ferent normative result (Ashley 2002). An advocate can alsoargue that the prior court was wrong, raising the interest-ing question of the precedent s continuing effect under thedoctrine ofstare decisis(Alexander 1989). Predicting the outcome of new Legal cases, on the otherhand, is fundamentally historical and empirical. Courtsmay have decided many more or less similar cases. Givena database of classified cases represented in terms of fea-tures that strengthen or weaken the classification, one canapply statistical or symbolic machine learning techniques toinduce general rules for classifying new cases and predictingtheir Outcomes . Even if these generalizations are rules, how-ever, they are not normative rules but empirical rules aboutnormative with Empirical Generalizationfrom CasesPrediction has not been studied to the extent argumenta-tion has in recent AI and Law research.

5 Empirical methodsfor Legal prediction pose a number of potential problems, andit remains an open question what limits constrain predictionas applied in classical approach for Predicting Outcomes employsstatistical or inductive models to generalize past cases. Thisprediction may be right from an empirical viewpoint, andyet be dead wrong from a normative one. A novel factualcircumstance in the new case may require a different out-come than the one predicted. General social, economic, ortechnological changes may call into question whole lines ofprecedents. Statistical generalizations, moreover, often haveexceptions. A prediction based on an empirical generaliza-tion of past cases may miss an analogy to an apparentlyanomalous precedent or counter-instance whose applicationmay, nevertheless, be supported by a convincing presents various other problems, as well.

6 Sta-tistical prediction of case Outcomes or feature weights mustdeal with the problem of small or biased samples. Evenin enormous on-line full-text Legal databases, only litigatedcases involving hard choices among evenly balanced conflict-ing strengths and weaknesses may be reported; many casesget settled and are not available for inclusion in the is an important question whether predictions based onsuch a biased sample will, for instance, be useful for , algorithms that rely solely on assigning quan-titative feature weights are problematic in so far as they arenot sensitive to a problem s particular context (Ashley &Rissland 1988) or do not support reasonable Legal explana-tions of why the predicted winner should win (a problem, forinstance, with (McKaay & Robillard 1974; Popple 1993)).

7 Finally, statistical algorithms require sufficiently large datasets, and as a rule of thumb, the harder the task, the morecases are needed. This adds an extra problem for predict-ing the outcome of Legal cases. The cases are texts andthe features must be represented in an appropriate formfor machine learning. So far, however, representation isa manual process (but see (Br uninghaus & Ashley 2001;Daniels & Rissland 1997)). Integrating Argumentation and PredictionNevertheless, AI models of Legal reasoning can play arole in predictive generalizations, when the system has adatabase of cases, represented by numerous factual and con-ceptual features that figure in their results ( , stereotypi-cal facts, Legal Arguments , or Legal conceptual analysis).

8 Most of these representations and case bases can be madethe basis of empirical predictions using general statisticaltechniques, machine learning algorithms, including, , neu-ral networks (Zeleznikow & Stranieri 1995), CBR methods,or specially designed algorithms that take some Legal knowl-edge into account. The predictions are subject, of course, tothe biases of the systems usually small and possibly unrep-resentative samplings, but they can be generated and theydo provide some information that may be relevant, even tothe Arguments the system questions arise therefore: How can, how should theargumentation and prediction tasks interact? Ideally, a cre-ative tension between argumentation and prediction may re-sult from combining normative and predictive knowledge inuseful ways.

9 For instance, analyzing induced empirical rulesfrom a normative viewpoint can reveal whether the prac-tical determinants of a result have little to do with whatthe normative rules say (Eisenberg & Henderson, Jr. 1992).Conversely, analyzing the normative rules from an empiri-cal viewpoint reveals whether they yield the right result asoften as lawmakers often combine normative and predictive in-formation. One cannot satisfactorily explain an empiricallegal generalization by simply listing the positive and nega-tive instances. Rather, one needs to recount normative argu-ments that could reasonably justify and explain the general-ization and its predicted result (Again, the normative coun-terarguments would be of interest.) Conversely, empiricalprediction can supplement the explanation of a normativeargument.

10 An empirical prediction that a Legal argument,though normatively justified, has little chance of success mayhelp one decide whether to pick a Legal argument or not, orwhether to settle and for how much (Waterman & Peterson1981). Finally, combining predictive and normative knowl-edge can be especially productive in improving paper describes a system that combines normativeand predictive information in a novel way. It does not usestatistics to generalize from a collection of cases. Instead itcombines a model of abstract Legal issues and their logicalrelations with a Case-based reasoning component. Becausethe model of the issues is central to the algorithm, it isnamed Issue- based Prediction (IBP). IBP identifies the is-sues raised in a case and uses a kind of scientific evidentialreasoning with cases to resolve conflicting evidence whenthe issue-related Factors favor both sides.