Research Evaluation Working Group
The INORMS Research Evaluation Working Group was established in 2018 to consider, over a two-year time-frame, how best to ensure that research evaluation is meaningful, responsible and effective. To this end, group members representing the research management societies that make up INORMS have decided to focus their attention on two main work-packages. The first seeks to draw attention to the burgeoning influence of World University Rankings on the behaviours of universities, and to consider how they might be made fairer and more responsible. The second seeks to develop some briefing materials aimed at senior university leaders, to appraise them of developments in the world of responsible research evaluation and how sensible evaluation approaches actually lead to better business decisions. They will report back on their activities through international conferences, national research management society communication channels and a discussion list set up to disseminate information about the project and solicit input and feedback.
If you would like to join the Working Group please contact Elizabeth in the first instance. If you would like to join the wider network please subscribe to the INORMS-RES-EVAL discussion list.
An article outlining some current research evaluation challenges and how the INORMS Working Group aims to address them is available at https://insights.uksg.org/articles/10.1629/uksg.456/
Rating the rankers
The Rankings sub-group have developed a mechanism for rating the World University Rankers, see Press Release 14.10.2020. The mechanism is based on a community-developed list of criteria for fair and responsible university rankings, translated into a series of questions by which each criterion might be judged. Six ranking agencies were invited to self-assess themselves against these criteria, after which a range of international experts formed their own assessments, calibrated by World Rankings Expert, Richard Holmes. The qualitative and quantitative data from the assessments have now been published. A blog post describing this work and the initial findings is now available.
SCOPE Framework for Research Evaluation
The Senior Manager Briefings sub-group have developed a five-stage process for evaluating responsibly, called ‘SCOPE’. The following resources are available on SCOPE:
- A blog post outlining the SCOPE process.
- A five-minute video introducing the SCOPE process.
- A one-page high-level overview of the SCOPE process. (Translations: Chinese, Danish, Icelandic, Norwegian, Brazilian Portuguese)
- SCOPE – a five-stage process for evaluating responsibly.
- A webinar on the SCOPE process for research managers was held on Monday 20 April 2020. To watch the recording, click here.
SCOPE Case Studies
The Briefings group has also created:
- Five arguments to persuade HE Leaders to evaluate research responsibly (Translations: Chinese, Danish, Icelandic, Norwegian, Brazilian Portuguese)
Any feedback you have on this work, including any local implementations of the SCOPE process, can be emailed to either INORMS-RES-EVAL@jiscmail.ac.uk (if you are a member) or directly to the Briefings Sub-Group leader, Laura Himanen, on email@example.com
Dr Elizabeth Gadd
Research Policy Manager, Loughborough University, UK
Elizabeth is the Champion of the UK’s Association of Research Managers and Administrators (ARMA) Research Evaluation Special Interest Group (SIG). She founded and chairs the Lis-Bibliometrics Forum and The Bibliomagician blog. She has a PhD in copyright and scholarly communication and does research into, publishes, blogs and speaks on, topics relating to research evaluation, bibliometrics, rights issues and open access. By day she works as the Research Policy Manager (Publications) at Loughborough University, UK!
Aline Pacifico Rodrigues
Manager, Research Support Office, Sociedade Beneficente Israelita Brasileira Hospital Albert Einstein, São Paulo, Brazil
Aline Pacifico holds a degree in Public Relations and a MBA degree in Strategic and Economic Management of Projects. She has been working for 16 years in the field of Research Administration in Healthcare Systems and Hospitals. Within this time she has been specialized in Pre-Award and Post-Award Administration, Assessing Research Performance and Structuring and Leading Research Support Offices. Today, she is Manager for the Research Support Office of Albert Einstein Jewish Institute for Education and Research.
Aline is also President of the Brazilian Research Administration and Management Association (BRAMA), organization founded in 2013 to enhance the professionalism of research manager within Brazilian universities, research institutes, funding agencies and other institutions which exercise the management of science, technology and innovation. She is also a member of the Society of Research Administrators International (SRA International) and holds a member representative role at International Network of Research Management Societies (INORMS).
Head of Department at the University of Iceland’s Division of Science and Innovation
Baldvin is head of department at the University of Iceland’s Division of Science and Innovation, where has been employed since 1999. He is involved in research evaluation of academic staff of Icelandic public universities, both yearly evaluation and evalution of applicants for academic positions and applicants for promotion. He has also been involved in development of the evaluation system and has been the contact point for the University of Iceland on university ranking.
Dr Baron Wolf
Assistant Vice President for Research Strategy & Data Analytics. Chief of Staff, Office of the Vice Present for Research, University of Kentucky
Dr. Wolf has more than 15 years of experience working in institutional research, effectiveness, and strategic planning. He has served on state and national boards and has presented at national conferences across the country on topics related to strategic planning, use of metrics, and data analytics. He holds a Bachelor of Science from Berea College, a Master’s in Public Administration from Eastern Kentucky University, and a PhD in Educational Policy Studies and Evaluation from the University of Kentucky.
Baron provides data analysis, strategic planning, advises on policy and program offerings. Works collaboratively with other research administration units on campus to provide robust business intelligence tools to faculty, staff, and leadership to improve decision making. Additionally, he serves as Chief of Staff and manages the day-to-day operations of the office. Dr. Wolf currently serves on the national advisory board for the University Benchmarking Project, a new national data collection in the United States aimed at collecting administrative unit data including student affairs, research administration, information technology, financial services, dining services, and many others. The project is a new effort of the National Higher Education Benchmarking Institute.
Dr Fang Xu
Associate Professor, Institutes of Science and Development, Chinese Academy of Sciences
Dr. Fang Xu holds a PhD degree in Management Science at University of Kent. She is now an associate professor of the Institutes of Science and Development, Chinese Academy of Sciences. She is also a member of Elsevier B.V. International Advisory Panel, the secretary general of Science and Technology Evaluation Special Committee of China Science and Technology Policy Studies, and the secretary general of Evaluation Method and Application Branch of Chinese Society of Optimization, Overall Planning and Economical Mathematics. Dr Xu’s research filed is S&T management and policy, and research evaluation. Dr. Xu has carried out long-term research on the theory and method of research evaluation, and has rich experience in organizing research institute evaluation , which has played a great role in supporting the macro management and decision-making of CASHQ.
Management Associate Professor, Tohoku University
After originally trained as an organic chemist at University of Tokyo and The Scripps Research Institute, I had two post – doctoral appointments at Harvard University and Kyoto University. Thereafter, taking advantage of a background of chemical research and international experience, I opened and managed the office of Royal Society of Chemistry (RSC) in Tokyo, Japan, for two years. The roles at RSC include promoting journals and building the network with the chemistry community in Japan. Since July, 2013, integrating all the experience and expertise in international, research, publishing and learned society experiences, I am contributing to the research support and promotion at Tohoku University, where 17,000 students are taught and instructed by 3,000 teaching faculty.
Manager, Research and Industry, The University of Melbourne, Australia
Justin Shearer is the Associate Director, Research Information and Engagement at the University of Melbourne, where he has portfolio responsibility for research digital scholarship, scholarly communications, and the University’s digitisation centre. Justin is also the Business Product Owner for the Find an Expert Refresh Project, which is tasked with developing a new researcher profile tool for the institution. He also is the leader of the Rankings Sub-Group of the INORMS-sponsored Responsible Evaluation Working Group where he represents the Australasian Research Management Society. Previously, he was Manager, Research and Industry in the Faculty of Business and Economics, where he had a focus on grant and industry funding. He has seven years’ experience in research management, with a particular interest in customer experience, business storytelling, industry engagement, researcher development and research evaluation.
Director of Research Analysis at the University of Guelph
Laura Beaupre is the Director, Research Analysis and Reporting in the Office of Research Services at the University of Guelph, a role she’s held since May 2017. In this role she: develops systems to evaluate and report on all aspects of the University’s performance; collects, synthesizes, curates, and analyzes research performance data from a variety of sources; and provides intelligence on research performance including grant successes, bibliometrics, and benchmarks.
Research Specialist, Tampere University, Finland
Laura is a Research Specialist working at the Research and Innovation Services at Tampere University. She is involved in the development, monitoring and reporting of research-related activities in accordance with the University’s strategy. She chairs the national expert panel for the culture of open scholarship and is a member of the Finn-ARMA executive committee. Laura’s expertise is in knowledge-based strategic planning. She is very familiar with different indicators of scientific productivity as well as methodologies of research assessment and university rankings.
Senior Advisor, Copenhagen University Library, Denmark
Marianne is coordinator of the bibliometric service at Copenhagen University Library. She is an advocate of open science, especially responsible metrics. Marianne is treasurer of Danish Association for Research Managers and Administrators (DARMA) and co-founder of the Danish Research Indicator Network (FIN). She holds a Master in Library and Information Science and a Master in Evaluation. Marianne’s research focuses on the construction of bibliometrics indicators and responsible research evaluation.
Senior Advisor, Department for Research and Development, Oslo Metropolitan University, Norway
Tanja Strøm is Deputy Chair of NARMA- Norwegian Network for Administration and Research Management, http://narma.no/. She led two Specialist Interest Groups (SIG): “Research Evaluation” and “Open Science”.
She is a senior adviser in the R&D department at Oslo Metropolitan University (OsloMet) where her main responsibilities are various aspects of Open Research (Open Science); publication strategies; making research visible, accessible and understandable; and evaluation of research. In addition, she gives lectures, supervises and teaches on these topics. She has developed services, infrastructure and policies for open access to research, and implemented this at OsloMet. She gives advice on Open Science policy topics.
She is a member of the INORMS Research Evaluation Working Group, https://inorms.net/activities/research-evaluation-working-group/. The working group is international and was established to find out how best to ensure that research evaluations are meaningful and responsible.
Tanja Strøm is a passionate advocate of transparency in science, open scientific communication and responsible research evaluation.
Prof Xiaoxuan Li
Researcher, Institutes of Science and Development, Chinese Academy of Sciences
Prof. Xiaoxuan Li is a researcher of the Institutes of Science and Development, Chinese Academy of Sciences (CAS). Dr. Li received his doctor’s degree from Peking University. He is now the director of the Management Innovation and Evaluation Research Center of the CAS; chairman of the S&T Management and Evaluation Professional Committee of China Science and Technology Policy Studies, chairman of the Evaluation Method and Application Branch of Chinese Society of Optimization, Overall Planning and Economical Mathematics; member of the advisory committee of DORA. Dr Li has rich experience in the field of science and technology policy, scientific research management, involving science and technology evaluation, human resource management, scientific research funds management, etc.
The group is currently being formed and representatives from all INORMS organisations are invited to join and help develop this important agenda. If you would be interested in participating, please contact Elizabeth directly at firstname.lastname@example.org.