definition of evaluation by different authors definition of evaluation by different authors

We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems. 2005). Impact is often the culmination of work within spanning research communities (Duryea et al. There is a great deal of interest in collating terms for impact and indicators of impact. The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. What is The Concept of Evaluation With its Importance? In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. It is acknowledged that one of the outcomes of developing new knowledge through research can be knowledge creep where new data or information becomes accepted and gets absorbed over time. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective. The process of evaluation involves figuring out how well the goals have been accomplished. To understand the method and routes by which research leads to impacts to maximize on the findings that come out of research and develop better ways of delivering impact. 2008; CAHS 2009; Spaapen et al. A Review of International Practice, HM Treasury, Department for Education and Skills, Department of Trade and Industry, Yes, Research can Inform Health Policy; But can we Bridge the Do-Knowing its been Done Gap?, Council for Industry and Higher Education, UK Innovation Research Centre. Its objective is to evaluate programs, improve program effectiveness, and influence programming decisions. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). 0000002868 00000 n 2009; Russell Group 2009). Capturing knowledge exchange events would greatly assist the linking of research with impact. The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. In undertaking excellent research, we anticipate that great things will come and as such one of the fundamental reasons for undertaking research is that we will generate and transform knowledge that will benefit society as a whole. Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). Standard approaches actively used in programme evaluation such as surveys, case studies, bibliometrics, econometrics and statistical analyses, content analysis, and expert judgment are each considered by some (Vonortas and Link, 2012) to have shortcomings when used to measure impacts. 0000004692 00000 n Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. 0000342937 00000 n A collation of several indicators of impact may be enough to convince that an impact has taken place. Two areas of research impact health and biomedical sciences and the social sciences have received particular attention in the literature by comparison with, for example, the arts. Explain. 2005; Wooding et al. 0000328114 00000 n Key features of the adapted criteria . It is therefore in an institutions interest to have a process by which all the necessary information is captured to enable a story to be developed in the absence of a researcher who may have left the employment of the institution. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. 0000008675 00000 n n.d.). 2009). There are standardized tests involved in the process of measurement assessment and evaluation to enable the students to make better use of the data available in the daily classroom. The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. Evaluation of impact is becoming increasingly important, both within the UK and internationally, and research and development into impact evaluation continues, for example, researchers at Brunel have developed the concept of depth and spread further into the Brunel Impact Device for Evaluation, which also assesses the degree of separation between research and impact (Scoble et al. 8. Definitions of Evaluation ( by different authors) According to Hanna- "The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students satisfaction. In this case, a specific definition may be required, for example, in the Research Excellence Framework (REF), Assessment framework and guidance on submissions (REF2014 2011b), which defines impact as, an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. These techniques have the potential to provide a transformation in data capture and impact assessment (Jones and Grant 2013). The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. Understand. In developing the UK REF, HEFCE commissioned a report, in 2009, from RAND to review international practice for assessing research impact and provide recommendations to inform the development of the REF. Evaluation is a process which is continuous as well as comprehensive and involves all the tasks of education and not merely tests, measurements, and examination. Evaluation of impact in terms of reach and significance allows all disciplines of research and types of impact to be assessed side-by-side (Scoble et al. 0000006922 00000 n Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. While defining the terminology used to understand impact and indicators will enable comparable data to be stored and shared between organizations, we would recommend that any categorization of impacts be flexible such that impacts arising from non-standard routes can be placed. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. evaluation of these different kinds of evaluands. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. What indicators, evidence, and impacts need to be captured within developing systems. The risk of relying on narratives to assess impact is that they often lack the evidence required to judge whether the research and impact are linked appropriately. These case studies were reviewed by expert panels and, as with the RQF, they found that it was possible to assess impact and develop impact profiles using the case study approach (REF2014 2010). This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. 0000001087 00000 n 2007). 2007). There are a couple of types of authorship to be aware of. This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. A comprehensive assessment of impact itself is not undertaken with SIAMPI, which make it a less-suitable method where showcasing the benefits of research is desirable or where this justification of funding based on impact is required. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. These . These metrics may be used in the UK to understand the benefits of research within academia and are often incorporated into the broader perspective of impact seen internationally, for example, within the Excellence in Research for Australia and using Star Metrics in the USA, in which quantitative measures are used to assess impact, for example, publications, citation, and research income. It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? HEFCE developed an initial methodology that was then tested through a pilot exercise. 0000342958 00000 n For systems to be able to capture a full range of systems, definitions and categories of impact need to be determined that can be incorporated into system development. Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. The development of tools and systems for assisting with impact evaluation would be very valuable. The Payback Framework systematically links research with the associated benefits (Scoble et al. Donovan (2011) asserts that there should be no disincentive for conducting basic research. , , . Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter. 0000001862 00000 n 0000348060 00000 n 0000010499 00000 n Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact. 0000007777 00000 n Enhancing Impact. stream (2006) on the impact arising from health research. Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. If impact is short-lived and has come and gone within an assessment period, how will it be viewed and considered? The Payback Framework is possibly the most widely used and adapted model for impact assessment (Wooding et al. At least, this is the function which it should perform for society. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. 5. Evaluation is a procedure that reviews a program critically. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. 2007). They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. Accountability. Different authors have different notions of educational evaluation. Measurement assessment and evaluation also enables educators to measure the skills, knowledge, beliefs, and attitude of the learners. 2007; Nason et al. To understand the socio-economic value of research and subsequently inform funding decisions. The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. For example, some of the key learnings from the evaluation of products and personnel often apply to the evaluation of programs and policies and vice versa. The Author 2013. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide, This PDF is available to Subscribers Only. Here we address types of evidence that need to be captured to enable an overview of impact to be developed. In education, the term assessment refers to the wide variety of methods or tools that educators use to evaluate, measure, and document the academic readiness, learning progress, skill acquisition, or educational needs of students. Such a framework should be not linear but recursive, including elements from contextual environments that influence and/or interact with various aspects of the system. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. Perhaps it is time for a generic guide based on types of impact rather than research discipline? The inherent technical disparities between the two different software packages and the adjustment . To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. In 200910, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) (REF2014 2010). Cb)5. 3. Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place.