Research Evaluation Working Group
The INORMS Research Evaluation Working Group was established in 2018 to consider, over a two-year time-frame, how best to ensure that research evaluation is meaningful, responsible and effective. To this end, group members representing the research management societies that make up INORMS have decided to focus their attention on two main work-packages. The first seeks to draw attention to the burgeoning influence of World University Rankings on the behaviours of universities, and to consider how they might be made fairer and more responsible. The second seeks to develop some briefing materials aimed at senior university leaders, to appraise them of developments in the world of responsible research evaluation and how sensible evaluation approaches actually lead to better business decisions. They will report back on their activities through international conferences, national research management society communication channels and a discussion list set up to disseminate information about the project and solicit input and feedback.
If you would like to join the Working Group please contact Elizabeth in the first instance. If you would like to join the wider network please subscribe to the INORMS-RES-EVAL discussion list.
An article outlining some current research evaluation challenges and how the INORMS Working Group aims to address them is available at https://insights.uksg.org/articles/10.1629/uksg.456/
Rating the rankers
The Rankings sub-group have published a list of criteria for fair and responsible university rankings and have developed a series of questions by which each criterion might be judged. Six ranking agencies are being invited to self-assess themselves against these criteria, after which a range of international experts will be forming their own assessments. We’re delighted that a World Rankings Expert, Richard Holmes, has agreed to calibrate the resulting ratings. Both qualitative and quantitative data from the assessments will be published when the process is completed. We hope this will provide a ‘proof of concept’ for future ranker evaluations. Any feedback you have on this work can be emailed to either INORMS-RES-EVAL@jiscmail.ac.uk (if you are a member) or directly to the Rankings Sub-Group leader, Justin Shearer, on email@example.com
Senior Manager Briefings
The Senior Manager Briefings sub-group have developed a five-stage process for evaluating responsibly, called ‘SCOPE’. The following resources are available on SCOPE:
- A blog post outlining the SCOPE process.
- A five-minute video introducing the SCOPE process.
- A one-page high-level overview of the SCOPE process.
- SCOPE – a five-stage process for evaluating responsibly.
- A webinar on the SCOPE process for research managers was held on Monday 20 April 2020. To watch the recording, click here.
The Briefings group has also created:
The ‘one-page high level overview of the SCOPE process’ and the ‘Five arguments to persuade HE leaders to evaluate research responsibly’ are currently being translated into various languages. To see translated versions click here.
Any feedback you have on this work, including any local implementations of the SCOPE process, can be emailed to either INORMS-RES-EVAL@jiscmail.ac.uk (if you are a member) or directly to the Rankings Sub-Group leader, Laura Himanen, on firstname.lastname@example.org
Dr Elizabeth Gadd
Research Policy Manager, Loughborough University, UK
Elizabeth is the Champion of the UK’s Association of Research Managers and Administrators (ARMA) Research Evaluation Special Interest Group (SIG). She founded and chairs the Lis-Bibliometrics Forum and The Bibliomagician blog. She has a PhD in copyright and scholarly communication and does research into, publishes, blogs and speaks on, topics relating to research evaluation, bibliometrics, rights issues and open access. By day she works as the Research Policy Manager (Publications) at Loughborough University, UK!
Aline Pacifico Rodrigues
Manager, Research Support Office, Sociedade Beneficente Israelita Brasileira Hospital Albert Einstein, São Paulo, Brazil
Aline Pacifico holds a degree in Public Relations and a MBA degree in Strategic and Economic Management of Projects. She has been working for 16 years in the field of Research Administration in Healthcare Systems and Hospitals. Within this time she has been specialized in Pre-Award and Post-Award Administration, Assessing Research Performance and Structuring and Leading Research Support Offices. Today, she is Manager for the Research Support Office of Albert Einstein Jewish Institute for Education and Research.
Aline is also President of the Brazilian Research Administration and Management Association (BRAMA), organization founded in 2013 to enhance the professionalism of research manager within Brazilian universities, research institutes, funding agencies and other institutions which exercise the management of science, technology and innovation. She is also a member of the Society of Research Administrators International (SRA International) and holds a member representative role at International Network of Research Management Societies (INORMS).
Head of Department at the University of Iceland’s Division of Science and Innovation
Baldvin is head of department at the University of Iceland’s Division of Science and Innovation, where has been employed since 1999. He is involved in research evaluation of academic staff of Icelandic public universities, both yearly evaluation and evalution of applicants for academic positions and applicants for promotion. He has also been involved in development of the evaluation system and has been the contact point for the University of Iceland on university ranking.
Dr Baron Wolf
Assistant Vice President for Research Strategy & Data Analytics. Chief of Staff, Office of the Vice Present for Research, University of Kentucky
Dr. Wolf has more than 15 years of experience working in institutional research, effectiveness, and strategic planning. He has served on state and national boards and has presented at national conferences across the country on topics related to strategic planning, use of metrics, and data analytics. He holds a Bachelor of Science from Berea College, a Master’s in Public Administration from Eastern Kentucky University, and a PhD in Educational Policy Studies and Evaluation from the University of Kentucky.
Baron provides data analysis, strategic planning, advises on policy and program offerings. Works collaboratively with other research administration units on campus to provide robust business intelligence tools to faculty, staff, and leadership to improve decision making. Additionally, he serves as Chief of Staff and manages the day-to-day operations of the office. Dr. Wolf currently serves on the national advisory board for the University Benchmarking Project, a new national data collection in the United States aimed at collecting administrative unit data including student affairs, research administration, information technology, financial services, dining services, and many others. The project is a new effort of the National Higher Education Benchmarking Institute.
Senior Manager, Research Programmes, UCD Research and Innovation, Dublin
Dipti Pandya is the Senior Manager and leads the UCD Research Programmes team. The team is responsible for the provision of the full range of pre-award supports for UCD researchers wishing to submit competitive proposals for national and international research funding. The team includes proposal support, researcher talent development and the EU Research Office. Key to this is the development and management of a strategic targeted approach to the research funding pipeline and researcher lifecycle. In 2018-19 over 1400 proposals were submitted with an annual research income of €125m in grants registered. Dipti is a member of the EARMA Policy and Representation Committee, CESEAR Taskforce for Competitive Funding and EU Advisory Group for Research Infrastructures.
Highly relevant to her current role, Dipti was the head of a funding agency as Director of the Irish Research Council, SSH National Delegate and Coordinator of the HERA and NORFACE ERA-NETs. Dipti has also acted as an independent consultant including writing a successful Enterprise Ireland-led MSCA Cofund (Career-FIT) and as an H2020 Expert Evaluator on 25 calls. In her career to date Dipti has been employed in a number of Irish HEIs including UCD, TCD, DCU, IADT and IT Tallaght.
Management Associate Professor, Tohoku University
After originally trained as an organic chemist at University of Tokyo and The Scripps Research Institute, I had two post – doctoral appointments at Harvard University and Kyoto University. Thereafter, taking advantage of a background of chemical research and international experience, I opened and managed the office of Royal Society of Chemistry (RSC) in Tokyo, Japan, for two years. The roles at RSC include promoting journals and building the network with the chemistry community in Japan. Since July, 2013, integrating all the experience and expertise in international, research, publishing and learned society experiences, I am contributing to the research support and promotion at Tohoku University, where 17,000 students are taught and instructed by 3,000 teaching faculty.
Manager, Research and Industry, The University of Melbourne, Australia
Justin Shearer is the Associate Director, Research Information and Engagement at the University of Melbourne, where he has portfolio responsibility for research digital scholarship, scholarly communications, and the University’s digitisation centre. Justin is also the Business Product Owner for the Find an Expert Refresh Project, which is tasked with developing a new researcher profile tool for the institution. He also is the leader of the Rankings Sub-Group of the INORMS-sponsored Responsible Evaluation Working Group where he represents the Australasian Research Management Society. Previously, he was Manager, Research and Industry in the Faculty of Business and Economics, where he had a focus on grant and industry funding. He has seven years’ experience in research management, with a particular interest in customer experience, business storytelling, industry engagement, researcher development and research evaluation.
Director of Research Analysis at the University of Guelph
Laura Beaupre is the Director, Research Analysis and Reporting in the Office of Research Services at the University of Guelph, a role she’s held since May 2017. In this role she: develops systems to evaluate and report on all aspects of the University’s performance; collects, synthesizes, curates, and analyzes research performance data from a variety of sources; and provides intelligence on research performance including grant successes, bibliometrics, and benchmarks.
Research Specialist, Tampere University, Finland
Laura is a Research Specialist working at the Research Development Services Unit at Tampere University. She is involved in the development, monitoring and reporting of research-related activities in accordance with the University’s strategy. To date, she’s been the project manager for three research assessment exercises, she chairs the Finnish working group for unifying the universities’ data submissions to international rankings and is an active member of the Finnish network for research support and administration (TUHA). Laura’s expertise is in knowledge-based strategic planning. She is very familiar with different indicators of scientific productivity as well as methodologies of research assessment and university rankings.
Dr. Lone Bredahl Jensen
Research Librarian, University of Southern Denmark, Denmark
Lone works as a research librarian at University Library of Southern Denmark, University of Southern Denmark, specializing in research services. Lone heads the library’s Responsible Conduct of Research (RCR) Theme Forum. In these capacities, Lone is involved in research policy development and she also coordinates and develops responsible conduct of research training for staff at SDU.
Lone holds a PhD in social science from the Aarhus School of Business (now Aarhus University). Lone has 20 per cent research time which she devotes to research analysis and integrity projects.
Lone is a member of the Danish Association of Research Managers and Administrators (DARMA).
Dr Patricia Smit
Research Innovation and Partnerships, Cape Peninsula University of Technology, South Africa
Patricia is a higher education specialist with a passion for people development, especially academics at the beginning of their careers. She joined the research management fraternity about a decade ago after an earlier career in the development field. From that background, social justice frames her practice. She recently joined the Research Directorate of the Cape Peninsula University of Technology as training manager and look forward to developing a few new initiatives, also in a regional context. She holds a PhD from the University of London (Institute of Education, now UCL IoE) and is a SARIMA Committee member.
Senior Advisor, Department for Research and Development, Oslo Metropolitan University, Norway
Tanja Strøm is a senior research adviser at Oslo Metropolitan University. She is part of the External funding team. The team is responsible for the provision of pre-award support for researchers writing competitive proposals for national and international research funding. The team includes researcher talent development and the EU-research office. Tanja is involved in the development, monitoring and reporting of research-related activities in accordance with the University’s strategy. She also provides advice on open science and the development of research publication strategies. She has a particular interest in researcher development and research evaluation.
Tanja is vice-chairman of the NARMA working committee and heads two Special Interest Groups (SIG): Bibliometrics and Open Science.
The group is currently being formed and representatives from all INORMS organisations are invited to join and help develop this important agenda. If you would be interested in participating, please contact Elizabeth directly at email@example.com.