What was the M17+ research evaluation about?

19 Mar 2021 Petr Kalíšek

M17+

The abbreviation may remind you of an arms industry product; however, it refers to an evaluation tool implemented in 2017, which was fully deployed in 2020. The introduction of the methodology means a change in the principles of national evaluation of research in the Czech Republic including more efficient and fair use of quantitative indicators and a greater focus on qualitative criteria. This aim is achieved through five modules:

• Module 1 – Quality of Selected Results: Selected results of the evaluated institution are assessed by an expert panel within the review procedure; processed by the Office of the Government.
• Module 2 – Research Performance: A set of monitored indicators concerning research output is evaluated; processed by the Office of the Government.
• Module 3 – Social Relevance: Evaluation focuses on applied research of organizational units of the evaluated institution and its benefits; processed by the evaluated institution.
• Module 4 – Viability: Various aspects of research management at the university level are evaluated (doctoral studies, HR policy, internationalization, drawdown of grants); processed by the evaluated institution.
• Module 5 – Strategy and Policies: Evaluation focuses on the institutional vision for the next reporting period; processed by the evaluated institution.

More information about M17+ (official title is Evaluating Research Organisations and Research, Development and Innovation Purpose-tied Aid Programmes According to M17+ Methodology) is to be found in the introductory article of em.muni.cz or on the MU website (both in Czech) providing detailed description of the methodology. What steps have been taken in practice?

Preparation and drawing up of the self-evaluation report

Masaryk University focused on the evaluation under M17+ even before it officially started. At the beginning of 2019, Monika Sieberová and Michal Petr of the RMU Research Office participated in a joint project with Charles University and Palacký University, which aimed at the assessment of possibilities of data collection in accordance with the MEYS criteria for the evaluation of Modules 3–5. This article primarily deals with the implementation of these three modules. One of the project outcomes was a feasibility study with conclusions for the MEYS. Thorough knowledge of the criteria definitely helped in the data collection.

The subject of M3–M5 evaluation was the period of years 2014–2018. It was officially launched in December 2019 by sending an application for authorization to organize the evaluation. Attached was the list of evaluators, who were addressed in the autumn 2019. Two changes in the composition of the panel followed and the final number of members was twelve plus a supervision from the MEYS. The evaluators were selected based on the following criteria: absence of publications prepared in cooperation with MU and no joint projects with MU in the past five years, a senior academic rank, work in the governing bodies of a faculty or university.

MU faculties started to draw up self-evaluation reports (“SE report”) for M3 in January 2020. The limiting number of characters for the respective M3 turned out to be a problem (only 7,200 characters per faculty). Basic information for M4 and M5 was prepared by the Rector’s Office. The SE report provided the main source of information for the international evaluation panel (“IEP”). When it was almost finished in late March, there was an outbreak of the pandemic and all milestones were postponed. The highlight of the evaluation process – an on-site visit – moved from May to November. Looking back, it is hard to imagine that the original schedule would be kept and all the logistics managed (accommodation, transportation, social events for evaluators etc.). Most likely, things would be less precise and the MU organizers would get less sleep.

Online procedure and evaluation report

The new reality affected by the closing of institutions and restriction of travel called for the cooperation with the evaluators held solely online. The SE report for M3–M5 including the summary of M1 and M2 was sent to the evaluators in May 2020. The IEP had almost four months for studying the reports. It sent questions regarding the content and form of the report to the University during that period, which were answered at an online session with the University management and representatives of the faculties in November 2020. Each faculty created a short introductory video recording for the evaluators to be able to better understand the ecosystems of the University and the city of Brno. The MU Rector’s Office prepared a website with instructions on the evaluation and work with supporting documents both for the evaluators and for MU participants.

A big event is happening on a small laptop display. An online International Evaluation Panel begins working with informal welcome call and opens the whole week evaluation of the @muni_cz within the frame of #metodika17. @MUNI_Science pic.twitter.com/32kxm3GjLi

— Michal Petr (@Michal_Petr_MU) November 9, 2020

After the four-day series of intensive consultations where all the panel members discussed the SE report of each faculty, a draft of the evaluation report (“ER”) was designed including grading based on the MEYS grade calibration (the weights of grades differ according to the discipline). The ER was provided to the MU management for comments in December 2020. After the evaluators collected the comments, they finalized the report and got it ready for dispatch to the MEYS. This was the final step of MU in the evaluation process. The MEYS now works on the consolidation and assessment of ERs of all universities.

Evaluation conclusions

The IEP praised MU for the organizational and technical background provided for the purpose of evaluation. The preparedness of the individual representatives of evaluated units and clear instructions contributed to the smooth negotiations with the faculties, which were conducted in Zoom with about 100 persons participating during four days. The evaluators expressed disagreement with the non-transparent calibration of the MEYS concerning final grades awarded for M3, which was based on the calculation using partial evaluation of eleven criteria at each evaluated faculty. The principle of determination of weights for the individual FORDs (OECD Fields of Research and Development), used for the classification of the respective MU faculties, was unclear. It would definitely be much easier for all the stakeholders to prepare the SE reports and the ER in a user-friendly web interface where texts could be edited, changes tracked, and links inserted.

The reports are now processed by the Ministry and the final results will be delivered to the University by the end of May 2021.

Application of the evaluation results

Both the faculties and the University may implement the conclusions of the evaluation report and the IEP’s discussions right away, since they were delivered in November 2020 (and their modified version in January 2021). The M3–M5 evaluation reports are a part of supporting documents for annual internal evaluations of faculties carried out in cooperation with the MU management in January and February.

The evaluation results form the basis for the adoption of measures in research management and thereby contribute to the University’s development. The recommendations will be integrated to the strategic documents of MU. The implemented measures and their effects will be the subject of next comprehensive evaluation in five years and the conclusions will be also utilized by the National Accreditation Bureau.

The evaluation results are also vital for MU’s funding, since they form supporting information on research costs for the proposals of the Research, Development and Innovation Council submitted to individual funding bodies (incl. the MEYS). Thus, the results of comprehensive evaluation (M1–M5) will form a basis for the determination of institutional support in the next five years.

Conclusion

The national evaluation in modules M3–M5 at Masaryk University has shown that the successful evaluation process needs long-term planning, careful preparation of supporting information and documents, as well as exceptional personal commitment of evaluators, representatives of the evaluated university and other stakeholders. Addressing the issues mentioned in this article would contribute to smoother organization of work in the future.

Research evaluation in general or within M17+ is definitely not a simple process involving just a few steps or variables. Despite its complexity, potential information overload and political pressure, it is a valuable tool, which must be governed by predetermined rules and maintained impartial. Then it is possible to reach conclusions that will help the academic community to become better informed and improve its position in the international context.

The author held the position of the secretary of the international evaluation panel during the preparation and the process itself. If you wish to read the view of an evaluation panel member, click here (in Czech).