Diagnostic Assessment for Improving Teaching Practice

1 International Journal of Information and Education Technology, Vol. 3, No. 6, December 2013. Diagnostic Assessment for Improving Teaching Practice Yuan Sun and Masayuki Suzuki . Abstract Most of current tests only report students' total II. COGNITIVE Diagnostic Assessment IN FRACTION. test scores, or T-scores, which are almost no use for providing PROBLEMS. Diagnostic information to inform teachers of their students'. strengths and weaknesses. Additionally, it has been shown that A. Cognitive Diagnostic Modeling most teachers have difficulty in using Assessment to improve their Teaching . Recently, cognitive Diagnostic Assessment (CDA) Most of current tests, which only report a small number of which is designed to measure specific knowledge structures and content-based subscores, total scores, or T-scores, are almost processing skills in students has attracted a great deal of no use for providing Diagnostic information to inform attentions.

2 In this paper, we apply a CDA approach in fraction teachers of their students strengths and weaknesses. In problems to 144 sixth grade students in an elementary school in recent years, cognitive Diagnostic Assessment (CDA) which Japan. We show how CDA can provide detailed information is designed to measure specific knowledge structures and about students' strengths and weaknesses and discuss the applicability of CDA for providing effective feedback for processing skills in students has been particularly attracting a teachers to improve their Teaching Practice . great deal of attention [7], [8]. CDA combines theories of cognition of interest with statistical models intended to make Index Terms Cognitive Diagnostic Assessment , attribute inferences about students mastery of so called attributes , mastery probability, effective feedback, fraction problems.

3 Which refer to students knowledge , cognitive processes , skills or strategies in a particular domain [9]. Through fine-grained Diagnostic reporting of students attribute I. INTRODUCTION mastery profiles, CDA provides more detailed information Assessment provides teachers with information about such concerning whether or not, or to what extent students have things as whether students have learned enough or what to mastered each of a group of specific defined attributes, rather teach them. Teachers are expected to use Assessment results than assigning each student only one single score. These to improve their Teaching and organize a sound instruction attributes are exactly what affect students learning plan for facilitating students educational development [1]. performance and their understanding and are exactly what Despite the importance of Assessment in education today, teachers need to help students to master.

4 Few teachers receive much formal systematic training in In a cognitive Diagnostic model, suppose there are k Assessment design or analysis [2]. As a result, teachers rely attributes in a particular domain, each student s mastery heavily on the Assessment offered by the publisher of their profile is formalized as a vector = ( 1, , k); k indicates textbooks or instructional material, or their own constructed the student s true ability along attribute k: k = 1 indicates classroom tests which are primarily used for assigning mastery of attribute k and k = 0 indicates nonmastery of this students grades. It has been pointed out that teachers attribute. Only certain attributes are measured by each item;. Assessment literacy is low and most teachers have difficulty information relating items to attributes is typically given by in using Assessment appropriately [3], [4].

5 To use Assessment the Q-matrix (item j attribute k) indicating which skills and to improve instruction and student learning, teachers need knowledge are involved in solving each item [9]. Letting j =. more training [5], [6]. Needless to say, it is important to plan 1, , J index the items in a test, the [j, k] entry of the and deliver training programs for enhancing teachers level Q-matrix (denoted qjk) is equal to one if item j measures of Assessment competencies. On the other hand, to develop attribute k; and zero otherwise. suitable Assessment tool to support teachers for Improving Attributes necessary for the domain need to be specified their Teaching Practice should be effective and important as first by experts in that area, by specifying what the key well. knowledge or skills are for students to understand the content In this study we aim to introduce cognitive Diagnostic in their learning process.

6 After that, test items and Q-matrix Assessment and its method for providing feedback effectively need to be constructed. Once the Q-matrix has been specified and show the Assessment has the potential to provide useful and items have been administered in a test, the items are information for teachers to improve their Teaching . Feedback calibrated to one of cognitive Diagnostic models. In this study is conceptualized here as information provided by an a DINA (deterministic input, noisy and gate) model [10], Assessment regarding aspects of students performance or [11] was applied in solving fraction problems. understanding. B. A Fraction Diagnostic Test We have developed eight attributes required in solving fraction problem according to the Japanese government Manuscript received April 25, 2013; revised June 30, 2013.

7 Curriculum guidelines for Teaching [12] (Table I) (see [13], The authors are with the National Institute of Informatics, 2-1-2. Hitotsubashi, Chiyoda-ku, Tokyo, Japan (e-mail: [14] for details). The numbers in the parentheses in the table denote grades where students are supposed to study the skills, DOI: 607. International Journal of Information and Education Technology, Vol. 3, No. 6, December 2013. according to the guidelines. attribute mastery profiles. Table III shows the four clusters . profiles, the sizes and mean scores of each cluster. TABLE I: EIGHT ATTRIBUTES DEFINED IN SOLVING FRACTION PROBLEMS. Attribute descriptions TABLE II: EXAMPLES OF FIVE STUDENTS ATTRIBUTE MASTERY. A1 understanding the meaning of fraction (3rd grade) PROBABILITIES WHO HAVE SAME NUMBER OF CORRECT ANSWERS. ID A1 A2 A3 A4 A5 A6 A7 A8.)

8 A2 changing fraction forms(4th). 1 A3 reducing to a common denominator (5th). 2 A4 reducing to the lowest denominator (5th). adding and subtracting fractions with a common 3 A5. denominator (3rd) 4 A6 multiplying (6th). 5 A7 dividing (6th). A8 deriving fractions from statements (3rd) 100%. A1. 100%. A1. A8 80% A2 80%. A8 A2. 60% 60%. We comprised a Diagnostic test of 35 items and 40%. 20%. 40%. 20%. administered it to 144 sixth grade students in an elementary A7 0% A3 A7 0% A3. school in Tokyo. A6 A4 A6 A4. A5 A5. ID=1 Average ID=2 Average III. RESULTS AND DISCUSSION. A1 A1. 100% 100%. A. Students' Individual Attribute Mastery Profiles A8 80%. 60%. A2 A8 80%. 60%. A2. Table II shows examples of five students attribute mastery 40%. 20%. 40%. 20%. A7 A3 A7 0% A3. probabilities estimated based on the DINA model who have 0%.

9 The same test scores of 22 (number of correct answers). We A6 A4 A6 A4. can see the student 2 masters A5 fully but does not master A4. at all. On the other hand, the student 3 masters A4 fully but A5. ID=4. A5. Average ID=3 Average does not master A5. A1. Although the five students have exactly the same test score, 100%. A8 80% A2. we can see they have different achievements in each attribute. 60%. 40%. The radar charts for their attribute mastery probabilities, 20%. A7 0% A3. which gives us a visual presentation more easily understood, are shown in Fig. 1. It was clear that students even with A6 A4. exactly the same test scores could have totally different attribute mastery profiles. These data cannot be obtained ID=5. A5. Average from the current typical educational tests. Fig. 1. Radar charts of the five students attribute mastery probabilities.

10 Each individual student s attribute mastery probabilities were reported which should be useful information for TABLE III: FOUR CLUSTERS WITH DIFFERENT TYPES OF ATTRIBUTE. MASTERY PROBABILITIES. teachers to know about each student s detailed knowledge state and give them appropriate guidance individually for Mean Attribute mastery probabilities Clusters scores A1 A2 A3 A4 A5 A6 A7 A8. their future remedial work. For example, for student 2, who does not master the skills of changing fractions to equivalent Cluster 1. fractions with a common denominator (A3) and reducing a (n = 96). fraction (A4), it might be effective to take them back to go Cluster 2. through again the meanings of the concept reduction to (n = 12). common denominator and reduction to the lowest fraction , Cluster 3. and support them to learn about the greatest common (n = 14).