Example: tourism industry

FORMATIVE AND SUMMATIVE EVALUATION

1 #euSMARTmap Sally Randles, Mohammad Hajhashem, Monica Gonzales, Eniko Demeny, Peter Kakuk Deliverable (D ) of the H2020 SMART-Map Project FORMATIVE AND SUMMATIVE EVALUATION FINAL REPORT 2 Contents 1. RATIONALE: The Role and Contribution of FORMATIVE / SUMMATIVE EVALUATION ..4 2. 3. FINDINGS ..43 4. DISCUSSION AND CONCLUSIONS ..55 5. 6. 7. List of Tables: Table 1 Definition of each actor group Table 2 Merged dataset: 6 countries, 3 technologies, and total analyzed by stakeholder category Table 3 Schedule of the SMART-Map Industrial Dialogues Table 4 Actor categories in the sample Table 5 Technology Table 6 De-facto responsible research by actor group Table 7 De-facto responsible research by technology area Table 8 De-facto responsible innovation by actor group Table 9 De-facto responsible innovation by technology area Table 10 Awareness and recall of RRI 3 Table 11 Awareness and recall of RRI by actor category Table 12 Awareness and recall of RRI by technology area Table 13 Awareness and recall of RRI by stakeholder group Table 14 Awareness and recall of RRI by technology area Table 15 Relevance of EC 5 k

Formative Evaluation is: “… a method of judging the worth of a program while the program activities are forming or happening” It comprises: “….evaluative activities undertaken during the design and pretesting of programs to guide the design process”

Tags:

  Evaluation, Formative, Evaluation formative

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Transcription of FORMATIVE AND SUMMATIVE EVALUATION

1 1 #euSMARTmap Sally Randles, Mohammad Hajhashem, Monica Gonzales, Eniko Demeny, Peter Kakuk Deliverable (D ) of the H2020 SMART-Map Project FORMATIVE AND SUMMATIVE EVALUATION FINAL REPORT 2 Contents 1. RATIONALE: The Role and Contribution of FORMATIVE / SUMMATIVE EVALUATION ..4 2. 3. FINDINGS ..43 4. DISCUSSION AND CONCLUSIONS ..55 5. 6. 7. List of Tables: Table 1 Definition of each actor group Table 2 Merged dataset: 6 countries, 3 technologies, and total analyzed by stakeholder category Table 3 Schedule of the SMART-Map Industrial Dialogues Table 4 Actor categories in the sample Table 5 Technology Table 6 De-facto responsible research by actor group Table 7 De-facto responsible research by technology area Table 8 De-facto responsible innovation by actor group Table 9 De-facto responsible innovation by technology area Table 10 Awareness and recall of RRI 3 Table 11 Awareness and recall of RRI by actor category Table 12 Awareness and recall of RRI by technology area Table 13 Awareness and recall of RRI by stakeholder group Table 14 Awareness and recall of RRI by technology area Table 15 Relevance of EC 5 keys of RRI by technology area Table 16 Benefits for individuals Table 17 Benefits for society by technology

2 Area Table 18 Challenges for individuals by technology area Table 19 Challenges for society by technology area Table 20 Challenges for society by actor group List of Figures: Figure 1 Potential Stakeholders for Precision Medicine (responsible) innovation ecosystem. Figure 2 Potential Stakeholders for Synthetic Biology (responsible) innovation ecosystem (UK). Figure 3 SMART-Map Stakeholder Breakdown Figure 4 Stakeholder Groups per Country Figure 5 Coalition of the Willing 4 1. RATIONALE: The Role and Contribution of FORMATIVE / SUMMATIVE EVALUATION A brief literature review. The establishment of a specific stream within the field of EVALUATION Studies seeking to provide definitional clarity between FORMATIVE VERSUS SUMMATIVE EVALUATION , reaches back at least 50 years. Early origins are attributed to Scriven (1967) working in the area of education performance.

3 Motivated by the desire to more effectively assess pupils learning and out-turn attainments, Scriven s foundational work provided a framework for discussing the philosophical and practical distinction between FORMATIVE and SUMMATIVE EVALUATION . Similarly, in the context of student performance in Universities, Murray (1980) says that FORMATIVE EVALUATION is developmental, with the primary objective being to positively support learning through longitudinal monitoring and systematic feed-back to the learners. Whilst SUMMATIVE EVALUATION by contrast, provides judgements by evaluating end-point performance against pre-set criteria and has a focus on supporting out-turn decision making. A comparison of SUMMATIVE and FORMATIVE assessment methods in education highlights the key differences1: SUMMATIVE Assessment FORMATIVE Assessment When At the end of a learning activity During a learning activity Goal To make a decision To improve learning Feedback Final judgement Return to the material Frame of Reference Sometimes normative (comparing a student against all others); sometimes criterion Always criterion (evaluating students according to the same critera) 1 Adapted from R.

4 Pregent, Charting your course: How to prepare to teach more effectively, Atwood, 2000. 5 In a different area, evaluating a new and experimental medical intervention, Stetler et al (2006), provide a useful commentary on the contexts that are best suited to FORMATIVE EVALUATION . The authors stress its appropriateness in situations where the project or programme being evaluated is novel, experimental, dynamic and unfolding. FORMATIVE EVALUATION helpfully accompanies such a process and is appropriate as a recursive (learn/evaluate/feedback/adapt/learn) approach. Its primary aim being to help the actors involved in the process to learn and adapt alongside the intervention. They say: FORMATIVE EVALUATION is: .. a method of judging the worth of a program while the program activities are forming or happening It comprises.

5 Evaluative activities undertaken during the design and pretesting of programs to guide the design process As an assessment method it focuses on: ..the internal dynamics and actual operations of a program in order to understand its strengths and weaknesses and changes that occur in it over time It is well suited to situations and research settings where the evaluators are seeking: i) Rapid-response learn and adjust processes during the length of the project ii) Understanding the nature and significance of the local implementation setting Stetler et al , 2006. 6 Today, the early origins of FORMATIVE and SUMMATIVE EVALUATION studies continue into contemporary scholarship and practice, with significant attention paid to education policy and practice at all levels, from Bloom s edited volume (1971) Handbook of FORMATIVE and SUMMATIVE EVALUATION of Student Learning; to Guyot (1978) who focuses on business education; to Murray (1980) and Harlen and James (1997).

6 Most recently in the education context, attention moves to different branches and disciplines within education, such as Brophy s Ed (2019) handbook of assessment, policy and practice in music education. Critical thinking on FORMATIVE and SUMMATIVE EVALUATION as an object of study ie evaluating EVALUATION , has extended from its origins in Education, to new policy areas such as medicine, health and social policy (Stetler et al 2006, Greve 2017). And yet, only relatively recently have issues of comparative EVALUATION and impact assessment been taken up to evaluate programmes in science, research and innovation policy (Edler et al eds., 2016). This extension, and the proliferation of applications to an ever-wider array of public policy areas, can likely be attributed to greater demand for evidence-based scrutiny of policy programmes, under the narrative of increased accountability to the tax-payer.

7 It sits against a backdrop of reductions in government spending at all policy levels: local, regional, national and international, and is accompanied by increased use of EVALUATION instruments at all levels. More recently, a powerful driver is the demand from policy audiences for evidence-based ex-post justification to support policy-spend in particular areas and defend budgets from being switched to alternative priorities. By demonstrating positive policy impact to the benefit of the full range of beneficiaries targeted under a particular instrument, these pressures can in-part be mitigated. This demonstrable-impact agenda has become a particularly acute policy imperative. Interestingly, the OECD Directorate of Education has also picked up the FORMATIVE / SUMMATIVE EVALUATION debate, producing a critique of the overly polarised representation of FORMATIVE / SUMMATIVE EVALUATION methods (Looney, 2011).

8 Looney argues for greater integration of FORMATIVE and SUMMATIVE elements within EVALUATION study designs. The author shifts attention to declare that FORMATIVE EVALUATION captures bottom-up learning whilst SUMMATIVE EVALUATION is appropriate for assessing system-level impact. SUMMATIVE EVALUATION addresses questions of accountability, value for money, and effectiveness in meeting the original policy objectives. Thus both FORMATIVE and SUMMATIVE approaches are 7 better considered in terms of their fitness for purpose under different project/programme/policy contexts and conditions. Here, the motivation and aims of the evaluators and the funders of the projects and programmes is an important consideration. Under this interpretation, FORMATIVE and SUMMATIVE elements can be combined or blended into study-designs, in a complementary rather than polemic fashion.

9 It is this more blended approach that characterises the SMART-Map EVALUATION . FORMATIVE and SUMMATIVE EVALUATION Applied to the SMART-map project. WP7 of the SMART-Map project implemented a blended FORMATIVE and SUMMATIVE EVALUATION process, designed to track the learning journey of the project across six countries , three emerging technologies (Precision Medicine, Synthetic Biology, and 3D biomed); and longitudinally across the full 30 months length of the SMART-Map project (1st May 2016 to31st October 2018). In line with the ethos of SUMMATIVE EVALUATION , the project s own objectives and goals provide the opening criterion against which to evaluate the project s performance and achievements: In addition, the EVALUATION team were keen to establish a Baseline position at the beginning of SMART-Maps project.

10 One important aim of the Baseline (quantitative) survey was to The SMART-Map goal: ..to define and implement concrete roadmaps for the responsible development of technologies and services in three key game-changing fields: precision medicine, synthetic biology and 3D printing in biomedicine. accessed 10 October 2018 8 capture respondents understandings and interpretation of responsible research and innovation before they engaged in the SMART-Map longitudinal process. Lets call this rri, or little rri as it captures respondents de-facto position on what responsibility in research and innovation means to them before they are exposed to the European Commission s definition of RRI through the instrument of the SMART-Map Industrial Dialogues (IDs). Let s call this EC policy definition of RRI, and indeed other policy-defined frameworks2 of Responsible Research and Innovation Big RRI.


Related search queries