What is ICT literacy, and why measure it?

1 11 ict literacy AssessmentInformation and communication TechnologyEducational Testing Service (ETS)Teresa Egan, Project Leader, New Product Development2 Topics What is ict literacy , and why measure it? The development of the ict literacy Assessment The current test design Sample tasks Score reports3 What is ICTliteracy, and whymeasure it?24 ict literacy the ability to use digital technology , communicationtools, and/or networks todefine access,manage,integrate,evaluate,create, andcommunicateinformation ethically and legally in order to function in aknowledge ModelICTL iteracyDefineAccessManageIntegrateEvalua teCreateCommunicateCognitiveEthicalTechn ical6 Built on prior efforts International ict literacy Panel, DigitalTransformation: A Framework for ict literacy (2002).

2 Association of College and Research Libraries(ACRL), information literacy CompetencyStandards for Higher Education (2000).37 ict literacy : a bridge betweenTechnicalLiteracyInformation LiteracyDatabaseWordProcessingPresentati on Can you bold a word? Can you open a database?AccessEvaluateUse Can you find information ? Can you evaluate authority? information and communication literacy Can you find information on the web? Can you create a persuasive presentation?8 Why measure ict literacy ? The nature, value, and availability of informationhas changed enormously, and this change impactsthe way we live, learn, and work.

3 To succeed, it is not enough to master technicalskills: you must know how to apply them withinan information society. There is a lack of information about the ICTliteracy of students, and debate about how best toaddress this issue in academic developmentof the ICTL iteracyAssessment410It started with our highereducation partnersCharter Clients California CommunityColleges California StateUniversity UCLA University of Louisville University of NorthAlabama University of Texas University ofWashingtonExpanded Consortium Arkansas StateUniversity Bowling Green StateUniversity Miami Dade College Oklahoma StateUniversity/DefenseAmmunition Center Portland StateUniversity Purdue University University of Memphis11 Development timeline Spring/Summer 2003

4 Higher education partnersidentified July, 2003 Initial definitional work began. December, 2003 Test development began withcharter client institutions. July, 2004 Field trials of tasks began. January, 2005 Operational version of large scaleassessment released for 6 months January April 2005: multiple institutions aretesting; goal is to test 6,000-7,000 students April May 2005: more market research May June 2005: analysis and score reporting;score reports sent to institutions June 2005: use analysis to finalize decisions re: Design for 2006 Large Scale Assessment andindividual reporting version Testing schedule (Fall and Spring?)

5 Just Spring?) Summer 2005: design workforce version of thetest513 The current testdesign14 The ict literacy Assessment Two separate sets of results: ict literacy Large Scale Assessment(2005) institutional profile only ict literacy Individual Assessment (2006) option toreceive feedback at individual student level Use in higher education Institutions: accreditation, state accountability,curriculum guidance Individuals: academic guidance, fulfillment of or tech. literacy requirement, certification15 Basic Design Features Interactive tasks using simulated software withlook and feel of typical applications (databases,search engines, spreadsheets, email, etc.

6 4-minute, 15-minute, and 30-minute tasks Real-life scenarios Get back on track mechanisms if test-taker getsreally lost Multiple scorable elements per task616 Assessment Content Length of tasks Long (30 minute) Medium (15 minutes) Short (4 minute) Contexts Academic Business Personal Subject Areas Humanities Social Sciences Natural Science Practical Affairs Popular Culture17 Delivery and Scoring Delivery Two-hour assessment window Web delivery at client institutions through securebrowser Scoring Automated scoring Proficiency estimation via IRT analysis18 Validity and reliability research Validity research.

7 Chain of validity method used during development to ensurecontent validity Extensive questionnaire before the test, to see how resultscorrelate with self-reported exposure to and confidence in specificproficiencies Questionnaire after the test to check face validity among test-takers Larger validity study in progress with charter clients Ongoing research planned Reliability efforts: Analysis of November field test Analysis of current 6,000-7,000 test-takers Ongoing scrutiny719 Sample tasks20 Sample Tasks Example 1: Comparing information Example 2: Display Data21 Comparing information Simple task: Four-minute duration Purpose: Assess the test-takers proficiency inintegrating information Integrate: The ability to interpret and representinformation in an ICT context.

8 This includes the abilityto synthesize, summarize, compare, and contrastinformation from multiple the task Summarize information from three different typesof sources Compare the information to reach a conclusion23In this task, examinees summarize information from avariety of sources and then draw conclusions from are presented with aninformation need and three different typesof information sources:9251. WebpageExaminees are presented with aninformation need and three different typesof information sources:262. EmailExaminees are presented with aninformation need and three different typesof information sources:273.

9 AdvertisementThe three sources present information according to differentconventions. A successful candidate must be able to locatethe relevant information in each are presented with aninformation need and three different typesof information sources:1028 Deciding how tocompare thesources involvesidentifying therequirements ofthe statedinformation filling in the table,examinees must interpretthe summary to rank thethree sources Data Simple task: Four-minute duration Purpose: Assess test-takers proficiency increating information Create: The ability to generate information byadapting, applying, designing, or inventing informationin ICT the task Visually represent data in a graph Interpret the graph to answer research questions32In this task, examinees create a visual representationof data to answer two research select whichvariables to display oneach of the the correcttime span involvesconsidering the implicitrequirements of theinformation the correctdependent variable (y-axis)involves thinking about howbest to reflect popularity.

10 36 Examinees have the opportunity to try out different graphsbefore settling on their response, and this process is factoredinto their the researchquestions involves correctlyinterpreting the two research questionsrequire different degrees ofanalytic reports1440 Score Reporting Score reporting will be online. Reports will come approximately eight weeksafter close of testing window, with reporting timereduced for subsequent administrations. Group comparisons if adequate Report Sample Performancelevels byproficiencypairings Explanatory textto guideinterpretation42 Sample of explanatory text -HighA High performance on ACCESS indicates that the studentsefficiently and effectively located information via browsing orsearching.