How do we measure research investment success?
Impact. It’s something that all research programmes need to demonstrate in order to access or increase funds. And to demonstrate impact, we need to put good quality measurements in place. In project management terms, this is often done with Key Performance Indicators across the project. So, for in-house commercial R&D projects, the measurement is relatively straightforward: how much have we invested in people, resources and time versus how much additional money are we making from new and upgraded products, or from IP licensing?
When two worlds collide
On the face of it, it may seem impossible to square the circle that is the commercial measurement mentioned above and academic measurement. For example, universities often measure the success of individual academic success as:
- How much research money do they bring into the institution, both private and public?
- How many publications and more importantly which journals are they published in?
Universities also use tools such as SciVal from Elsevier, or Scopus, which holds all academic papers. This gives a ‘field weighted citation’ which is a measure of how ‘good’ the academic is.
The government’s method of assessing the quality of research is the Research Excellence Framework. The outcomes of these assessments inform the allocation of public funding for UK universities, which is currently at around £2 billion annually. In order to be assessed, UK universities must submit case studies on the impact of their research – this can be research just within the university or alongside commercial collaborators.
Universities also get measured on their teaching quality – through the Teaching Excellence Framework – and will be part of the new Knowledge Exchange Framework when it is fully launched. This new framework is designed to measure knowledge exchange, and it is intended to give commercial organisations deeper information on which to base their partnering decisions.
What to think about
When it comes to long-term research projects where both commercial and academic measures of success need to be met, there are some key considerations:
- How can we make sure we meet our objectives in the same time frame?
- How can we work together to ensure that everyone understands both the need to publish and the timeframe for negotiations on commercial IP?
- Is there an understanding from the commercial organisation that a ‘win’ for the university isn’t necessarily just answering the company challenge with one piece of research. It may have to be broken down into a jigsaw and each piece will need to get funding, which can be time-consuming.
- Does the university understand its commercial partner? Can they work together to bring the right skills, facilities and access to resources together?
- Is there a safe place to challenge and test ideas? Are both parties committed to diversity of thought and a transparent process?
And this might translate into conversations that look like this:
- What does success look like to the grant or funding body we are working with? What do we have to report?
- Is everyone agreed on the project’s goals? It’s impossible to measure if there’s no agreement on outcome.
- Who is responsible for measurement, and how often should we report during the project?
- Can we agree on key metrics at the start of the project? This should be part of the scope and goal-setting stages.
Case study – Rolls Royce University Technology Centres
For Rolls Royce, a key question is: “What if the industry didn’t have to wait for graduation in order to start making the most of universities?’ Instead of just using standard graduate schemes, why not start things earlier?
Rolls Royce’s University Technology Centres (UTCs) are spread across 31 institutions globally, with each centre prescribed to address a key technology. Formalised, long-term partnerships with the UTCs equip Rolls-Royce with efficient access to high-quality research, allow the development of associations with lecturers and course deliverers. They connect the company with the wider academic world and provide a mechanism for training the next generation of experts.
Crucially Rolls-Royce recognises the advantage of being able to apply real-life content to degrees in order to make them more readily transferable to the working world. And yet, this can create friction between companies and universities – precisely because there are different metrics for success.
In her report to the government in 2015, Professor Dame Ann Dowling suggested that industrial placements and things that have a real-life ‘impact’ should be weighted equally alongside all other university coursework. Dowling’s report, which generally reasons the benefits of short-term and long-term knowledge exchange, hammers home the need to invest in collaborative research and development as a driver for overall national growth and productivity. Rolls Royce agrees, and so measures the success of its UTCs by the number of graduates or PhD students who end up working for the business or in its supply chain, and by the contribution, the research makes to Rolls Royces’ commercial success.
To contextualise the success of the UTC network, consider that around 25 per cent of the PhD graduates involved in it go on to work directly for Rolls-Royce, the project sponsor. This is much better than the industry average, with more still joining the supply chain. Those who choose to stay within academia continue to contribute research to industry well after completing their courses and Rolls-Royce makes sure that it sustains this relationship effectively as well.
It’s fair to say that measuring the success of research projects will depend on the project, the parties involved, the nature of the relationships and the desired outcomes. At a time when we need to be able to demonstrate impact both academically and commercially, this is an area of R&D project management that cannot be neglected.