DEFINING IMPACT

SCHOLARLY IMPACT

Section Contents:

Introduction

• 
  Rationale
  
• 
  Education Scholarship
  

Defining Impact

• 
  What is Impact?
  
• 
  Development in research metrics
  
• 
  Grey Metrics
  
• 
  Critically Examining Metrics
  
• 
  Activity 1
  

Telling Impact Stories

• 
  Impact Stories
  
• 
  Activity 2
  

Moving Forward

Critically examining metrics

We should not privilege research over other ES activities, which the current 3P criteria does

All about promotion!!!! https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6325612/

Peer-review issues: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1383755/

Reflecting on our purpose and goals as educators and education researchers can help us ensure we are not only chasing numbers, but This will affect how we practice and what we choose to do. TO NOT only chase numbers, but reflect more on why/how we’re really accomplishing our goals as educators/education researchers.

-tension b/t what we want to do/hope to achieve with practicalities (impact agenda). we don’t always do what we want to, but might tailor to whatever is getting funded now b/c popular (e.g. CanMEDS, humanism in med ed, patient-centred care – these are our incentives).

-DEFINITION OF IMPACT AGENDA from Anderson 2015 “‘Impact Agenda” and operationalized through the audit and performance management systems of the higher education sector (Rogers, Bear, Hunt, Mills, & Sandover, 2014)”  questions related to who controls and carries out impactful research, and who the beneficiaries of such research will be, are rarely considered.

Dora, Metric Tide, responsible use of metrics

“Somehow, academic numbers have come to count as something ‘tangible’- even if the value of such numbers remains a mystery. An article has been cited a given number of times. Has it been understood? Is its argument profound or even correct? Sponsored research increases. But is the work it funds valuable, or does it merely contribute to the mountain of information under which we are buried, the creation of trivial consumer items, or to the growing inequality of American society?” (Barr Frodeman)

Not just implications for criteria we use for promotions, tenure, grants, but also publication & authorship – new OA systems, but how about issues with peer review

The metrics we set out have implications for faculty behaviour, faculty who role model certain ways of being, which affect the next generations of education scholars that they are teaching/mentoring/working with. Metrics aren’t a hidden agenda, they are an explicit agenda. What does the explicit agenda say about what we value and want to replicate (all education is reproduction)?

What makes an education project scholarly?

CAME uses the “3p” criteria to determine whether an education project is scholar (i.e. is, in fact, education scholarship). To be scholarly, an education project must be:

•Peer-reviewed
•Publically disseminated
•Provides a platform that others can build on

The 3p criteria raises some issues, which will be addressed in the IMPACT SECTION.

both research and innovation is important since it expands our consideration of what can be ‘counted’ as legitimate academic work. As such it allows us to explicitly evaluate and support both traditional research in education and accomplishments, e.g., publications in peer-reviewed journals, as well as non-traditional academic activities, e.g., the design, implementation and evaluation of an educational innovation, e.g., a new teaching strategy, tool, programme or curriculum”

PAGE 13: With respect to evaluation, it is important to go beyond traditional academic considerations. For example, the quality of “peer review” is most often equated with the number of publications in a “high impact”, peer- reviewed journal. The inclusion of educational innovations however requires us to acknowledge the importance of peer reviewed journals related to health professions education and that their related impact factors may need to be assessed differently. It is also important to evaluate impact in relation to non-traditional avenues of dissemination (e.g., demonstrating the adoption of a peer reviewed innovation in a way that impacts on practice). The challenge in “public dissemination” is to also consider expanded avenues for dissemination. For example, the dissemination of educational innovations can include web-based repositories, technical reports and/or other non-traditional venues for the sharing of innovations.

Previous

Next