KEY INDICATORS OF INNOVATION PERFORMANCE: PERCEPTION OF SIGNIFICANCE AND PRACTICAL APPLICATION

Introduction

The introduction of innovations as a source of competitiveness between industrial companies contributes to the fact that the issues of assessing the effectiveness of innovative activities attract more and more attention among researchers and experts [Klein et al., 2001; Glassman, 2009; Trachuk, Tarasov, 2015]. At the same time, a number of specialists consider the effectiveness of innovation activity as one of the most significant [Dennis, 2003; Morris, 2008; Glassman, 2009].

It should be noted that despite a fairly developed classification of innovation metrics, they were all developed by foreign researchers, and the possibility of their application in the practice of Russian industrial companies has not been sufficiently studied. There are no empirical studies which analyze the use of metrics by Russian industrial companies. An overview of the existing classifications related to innovation performance metrics is given by the authors in the articles [Trachuk, Linder, 2016; 2019; Trachuk and Linder, 2019].

The studies of foreign authors on the use of key performance indicators show that investments in innovative technologies, such as lean manufacturing, customer relationship management (CRM), enterprise resource planning (ERP) and rapid prototyping cannot successfully complete the implementation stage and evaluate improvements [ Dennis, 2003] as efforts on implementation must be monitored and diagnosed [Shiba et al., 1993].

In addition, evaluating the effectiveness of innovations provides an opportunity to identify problems and weaknesses and to take corrective action before these problems escalate [Kueng, 2000].

The purpose of this study is to analyze the perception and actual application of key performance indicators of innovative activities by industrial companies.

The choice of industry is explained by several reasons. First, it is industrial companies that have the largest investments in innovation. Secondly, during an economic downturn and limited access to financial capital, industrial companies will assess the effectiveness of investment in innovation more closely. Thirdly, due to the peculiarities of the industrial sector, justifying the costs of innovation activities through the introduction of key performance indicators is one of the highest priority tasks.

To achieve this goal, a task was set based on the classification of innovations and innovative metrics existing in the academic literature, which were described by the authors earlier [Trachuk, Linder, 2019;], to study which of them are recognized by practitioners as the most significant and which of them are most often used in practice. As part of the empirical approach, in-depth interviews and questionnaires with representatives of the top management of Russian industrial companies were conducted.

1. Theoretical review and formulation of research hypotheses

In academic research, there are several areas for assessing the effectiveness of innovation.

The earliest one was performance evaluation based on financial indicators such as return on investment [Norman, Bahiri, 1972; Barbosa, 2004]. However, later this approach was criticized, since, according to researchers, it does not provide feedback on the effectiveness of internal business processes and the contribution of employees to achieving this efficiency [Kaplan, Norton, 1992; Abdel-Maksound, 2004].

Therefore, in the future, researchers began to offer indicators that measure the innovation process and innovative activity of companies, for example [Morris, 2008].

However, no single performance indicator can capture the full complexity of innovation activity [Amaratunga et al., 2001]. To be successful in today's globally competitive environment, organizations must be able to capture objective (e.g. unit costs, profit) and subjective (e.g. quality, satisfaction) measures. Therefore, researchers began to propose models that allow for the evaluation of innovative activity in a complex way [Coombs et al., 1996; Verhaeghe and Kfir, 2002; Gupta, 2007; Ortiz et al., 2007].

Other researchers began to use econometric modeling to analyze the relationship between innovation and company performance [Lööf, Heshmati, 2002; Mairesse and Mohnen, 2004].

A number of other authors have proposed the use of multidimensional indicators [Salomo et al., 2008; Good, Tellis, 2009; Choi, Ko, 2010], as well as to investigate the effectiveness of balancing financial and non-financial indicators in the performance evaluation system [Kerssens van Drongelen, Bilderbeek, 1999; Hudson et al., 2001; Kanji and Sa', 2002; Savioz, Blum, 2002; Bremser and Barsky, 2004].

In addition, some studies have emerged that focus on the use of financial and non-financial indicators by companies and assessment of their perceived importance. The results of such studies have led to the conclusion that there is a direct relationship between a company's assessment of the perceived significance of an indicator and its actual use, for example [Kaplan, Norton, 1993; Cox et al., 2003].

A variety of approaches to measuring the effectiveness of innovation activity is presented in Table 1.

Thus, the considered approaches can be divided into the following groups depending on their focus:

1) related to the innovation process;

2) indicators grouped in a balanced scorecard;

3) other approaches, for example in relation to the innovation climate or the ability to innovate.

Models of the first group use the idea of measuring the "input – process – output" of innovation activity [Goffin, Mitchell, 2010; Tidd and Bessant, 2014]. Some authors, in particular [Möller et al., 2011], propose to separate the concepts of “output” as a short-term effect and “result” as a long-term effect, as well as the system for obtaining the result (for example, marketing and sales efforts) and the result itself [Vahs and Brem, 2015]. Thus, the Input – Process – Output – Outcome (IPOO) model uses specific indicators for each stage of the company's innovation process, such as the training cost of one employee at the entry stage and the number of new products at the exit stage [Fischer et al., 2015].

It should be noted that the use of outcome criteria such as market share gained through innovation, etc., entails two distortions:

– firstly, one cannot be sure of obtaining reliable measurement results, since a sufficiently large amount of time can pass between the introduction of an innovation and its positive effect;

– secondly, there is a difficulty in clear distinguishing the effect of innovation from other factors that could lead to a measured increase in a market share.

Along with complex approaches to measuring the innovation process, there are approaches in the research that focus on certain aspects of the process. For example, the R&D Return system [Vahs, Brem, 2015] was created to evaluate performance in R&D only. In this approach, R&D productivity is measured by the potential productivity and efficiency of technology development and compared with the R&D profitability which consists of potential productivity and operational efficiency. In addition, to calculate the value of the return on R&D, a certain algorithm is used that combines all indicators. This system can be transferred to other business units of the organization.

In contrast to these scoring systems, Bean consulting company proposed to simply quantify the innovative output of the R&D department [Pappas, Remer, 1985]: the number of patents, publications, citations, etc. or simply by counting the number of new ideas [Fuchs, 2014].

The second group of approaches includes those based on a balanced scorecard, such as the Innovation Balanced Score Card [Fischer et al., 2015]. The four dimensions of a typical balanced scorecard (finance, customers, internal business processes, learning, and growth [Kaplan, Norton, 1992]) are assessed with the help of a company's innovation strategy with a focus on improving the effectiveness of innovation, such as time to market for a new product, market share obtained through R&D [Žižlavský, 2016].

Another assessment method is innovation audits. When conducting an innovation audit, not only the results of innovative activities (output indicator) are considered, but also how these results were achieved (process indicator). For this, aspects such as innovation strategy, market, product, technology, etc. are evaluated. [Goffin, Mitchell, 2010]. Examples of such indicators for the “market” aspects are: change in market share thanks to new products, number of customer surveys, number of implemented innovations based on customer ideas, etc. [Warschat, 2005] or 5 measurements of A.T. Kearney "House of Innovation" or Improve [Innovety, 2014].

The group of other approaches is diverse. They usually have a specific focus, such as INNO assessing the innovation climate [Kauffeld et al., 2004] or describing general determinants of innovation ability [Jong et al., 2001]. Some of the used criteria intersect with other approaches, some are specially adapted to a specific task (for example, export activity) [Jong et al., 2001].

Thus, there is a wide variety of metrics to measure innovation in companies or business units. They vary widely in complexity and data requirements involving effort to collect and analyze data. In choosing the right approach for a particular situation, the differentiating criterion should probably be the availability of the resources necessary to collect and analyze the data.

Separately, it is possible to single out the indicators used for the operational control of innovation activity focused on the indicators for the implementation of innovative projects [Fischer et al., 2015].

Similar to the due diligence method, indicators of typical project control are proposed, such as control of project stages or budget [Maier et al., 2007; Vahs and Brem, 2015]. In other studies, the effectiveness of innovative projects is measured by the duration of a new product development and its costs [Fuchs, 2014] or the return on investment of innovative projects [Hauschildt, Salomo, 2007]. In addition, some studies argue that the only way to assess innovation activity is to evaluate the progress of these innovation projects [Littkemann, Derfuß, 2011]. However, this argument ignores the fact that innovation in a company is not always associated with projects, but the whole company including project activities, teamwork, should strive to be innovative [Fischer et al., 2015]. Thus, in addition to measuring the effectiveness of innovative project implementation, indicators are necessary to evaluate the achievements of innovative teams.

The purpose of this study is to analyze which of innovation efficiency indicators are used by Russian industrial companies, which of them are perceived by management as significant, and whether the use of the indicator depends on the recognition of its significance for the success of innovation. In addition, it is interesting to answer the question how much the implementation of an innovation strategy and the company's adherence to one or another type of innovative behavior will affect the perceived significance of indicators for assessing the effectiveness of innovative activity and their use.

The hypotheses of this study are:

H1. The perceived importance of KPIs will differ depending on the type of innovative behavior an industrial enterprise pursues: radical innovators, technological innovators, efficient producers, value-innovators, or imitators.

H2. For technological and radical innovators, non-financial performance indicators of innovation activity will be the most significant, and for creators of value innovations, efficient producers and imitators, financial performance indicators of innovation activity will play a great role.

H3. For companies of all innovation regimes, there will be a positive relationship between the perceived importance of the indicator and its actual use in innovation performance assessing.

2. Methodology of the study and description of the sample

Since the design of an exploratory study was chosen to achieve the goal of the work, in-depth interviews became the main method of collecting information at the first stage. At the second stage, a survey of top management representatives of industrial companies was used. The methodology of the first and second stages of the study is presented in Table 2.

Since the design of the study involves the participation of one to five employees from each company, a total of 37 respondents were interviewed – heads of departments involved in innovation activities (in particular, respondents held the positions of heads of R&D departments, heads of research institutes, strategic planning departments, etc.), as well as deputy top managers for innovation, science and development. Interviews took place from October to November 2021 in person or via videoconference. The duration varied from 30 minutes to 1 hour. The average interview time was 45 minutes.

At the second stage, 589 questionnaires were sent to representatives of Russian industrial companies. Feedback was received from 139 companies, the response accounted for 23.6%. The high response rate is explained by the fact that some of the questionnaires were distributed among students on additional professional education programs from the Faculty of Higher School of Management of the Financial University, where the authors deliver lectures. The survey took place from December to February 2022.

3. Formation of the study sample

When forming the sample, the following criteria for selecting companies were determined. Firstly, the company must belong to the manufacturing industries and operate on the territory of the Russian Federation. Secondly, the company must be engaged in innovative activities, and thirdly, the number of employees of the company must be more than 100.

Since the hypotheses of the study require a breakdown of the set of enterprises according to the type of innovative behavior, then they were further broken down in accordance with the characteristics inherent in a particular type of innovative behavior, described in the article [Linder, 2020] and presented in Table 3.

The distribution of companies by the type of innovative behavior is presented in Table 4. The characteristics of the companies included in the sample are presented in Table 5. It should be noted that the R&D expenses of the sample companies are characterized by an extremely wide spread.

4. The results of the empirical study

All in-depth interviews were analyzed with the use of the content analysis method and checked for emic authenticity. As a result, 63 innovative metrics used by Russian industrial companies were identified and divided into 8 groups.

Table 6 presents the obtained results.

As can be seen from Table 6, Russian industrial companies most often use metrics evaluating commercialization (94%), economic effect (97%) and focus on technological innovation (78%) to assess the effectiveness of innovation.

The least frequently used metrics are customer focus (28%), intercompany collaboration (47%) and the key role of the CEO (38%).

However, it should be noted that such a distribution of metrics is not typical for all companies. The importance of certain metrics depends on what type of innovative behavior the company adheres to (Table 7).

According to the companies belonging to radical and technological innovators innovation activity, on the one hand, should have a positive effect on the results of research and development (patents, prototyping, etc.) and, consequently, on commercialization and economic efficiency, expressed in increasing productivity, cost reduction as a result of the introduction of technologies, etc. On the other hand, it is not so much the economic and financial result that is important, but the overall level of innovation activity: the amount of investment in non-key innovation projects, the level of spending on the development of fundamentally new products and services, etc.

At the same time, for the creators of value-added innovations, customer focus is important in innovative activities, which allows them to commercialize innovations effectively and, consequently, obtain a positive economic effect.

Thus, our first hypothesis is confirmed: indeed, companies that implement different types of innovative behavior consider different groups of performance indicators important for themselves.

To test the second hypothesis, respondents were asked whether they consider these indicators important for their organization, and whether the organization uses them to measure the effectiveness of innovations. Since some respondents may report using metrics to measure the effectiveness of innovations whereas they do not actually use them, we applied a methodology designed to reduce the “social desirability” effect. That is, to ensure accuracy in responses, participants were asked to indicate the actual measurement systems used to collect information for each indicator. For example, respondents who reported that they measured the innovation activity level of employees engaged in innovation work should describe a measurement recording system, such as an employee survey, as the tool used to obtain this information. The results are shown in Figures 1–5.

Differences in the recognition of innovation metrics importance and their use in the company were tested by the ANOVA method to identify statistically significant differences. It turned out that the recognition of the importance and use of innovative metrics vary greatly across groups of companies, depending on the economic behavior under implementation.

It was found out that the perceived importance and actual use of innovation metrics are interrelated for companies of all innovation modes: radical innovators – Pearson's – χ² (25, 34) = 54.27, p < 0.001; technological innovators – Pearson's – χ²(17, 55) = 67.17, p < 0.000; efficient producers – Pearson's – χ² (31, 64) = 58.73, p < 0.000; создатели ценностных инноваций – χ² creators of value innovations – Pearson's (29, 71) = 69.78, p < 0.000; imitators – Pearson's – χ² (26, 59) = 66.15, p < 0.000 respectively.

It was also obtained that the perceived importance and actual use of financial metrics is higher for efficient producers, value innovators, and imitators, while the perceived importance and actual use of non-financial metrics is higher for technology and radical innovators.

Thus, the second hypothesis confirms that for technological and radical innovators the most significant will be non-financial indicators of the innovation effectiveness, and for the creators of value-based innovations, efficient producers and imitators, the most significant will be financial indicators of the innovation effectiveness.

However, the overall results show that 38-40% of technology and radical innovator companies that highly rated the importance of non-financial KPIs (i.e. quality or product/service measures, new product creation time, employee development and knowledge) did not use these key performance indicators to evaluate the effectiveness of innovation activities. Similarly, 21% of value-innovating companies that highly rated customer satisfaction did not use this KPI.

As a result, it should be recognized that there may be some potential barriers between the perception and actual use of both financial and non-financial indicators.

Thus, the third hypothesis was partially confirmed.

5. Discussion of the results of the study

It should be noted that recently industrial companies have been paying a lot of attention to both product and process innovations, which makes it necessary to evaluate the effectiveness of such activities. At the same time, the variety of innovative activities allows companies to calculate not only the main metrics, but also more complex ones.

It is important to note that when distributing metrics into groups, some indicators, depending on the purpose of application, can be attributed to other groups. For example, the metric "Costs for the development of a new product" can be attributed both to the level of innovative activity and to its effectiveness.

Frequency analysis showed that seven metrics are used by more than 50% of companies, the remaining metrics are used by no more than 15% of companies.

In addition, the importance of certain metrics depends on what type of innovative behavior the company adheres to (Table 8).

In most cases, industrial companies use well-studied standard metrics, adapting them in accordance with the goals set in the innovation strategy, that is, in accordance with innovative behavior. At the same time, about 10% of companies use unique, specially designed innovation metrics. Most of these companies belonged to the type of innovative behavior "technological and radical innovator". Also, for companies of these types, non-financial metrics for assessing the effectiveness of innovation activity turned out to be more significant than financial ones. For companies of the other three innovative types of behavior, financial metrics are more significant. Among leading and lagging indicators, the second ones are used to a greater extent, allowing to judge the achievement of goals for a certain period of time.

In most cases, the recognition of a metric as important entails its use to assess the effectiveness of innovative activities. However, this trend is not typical for all indicators.

For example, during the interviews, metrics reflecting collaboration in innovation activities were recognized as important (especially by companies classified as technological innovators and efficient producers), but subsequently they were rarely used to assess the effectiveness of innovation activities.

You can also notice that the more innovative activity is developed in a company, the more unique metrics are used to evaluate its performance.

6. Conclusions and limitations

The study showed that despite the universality of innovation activity metrics they are used in accordance with the objectives of the innovation strategy and the type of innovative behavior of an industrial company. In addition, the study showed the importance of the metric perception by the head of a company, since the majority use those metrics that are perceived by the head as the most significant. More often, financial metrics and lagging metrics are used to assess the achievement of goals over a certain period of time.

Based on the results of this study, it can be assumed that raising the awareness of managers about the balance of financial and non-financial indicators will increase the actual use of these indicators.

Since well-studied standard metrics were used more often, conducting trainings, seminars or discussion panels on the use of innovation performance metrics will allow business leaders to learn more about what kind of metrics are used, increase awareness of the importance of financial and non-financial indicators for assessing the effectiveness of innovation activity.

This empirical study was qualitative, which causes its main limitation. In-depth interviews as a qualitative method of collecting information are not representative, so their results cannot be extended to the entire population of industrial companies in Russia. In addition, the industry is quite diverse, so it is not possible to determine to what extent the surveyed companies are typical and characteristic of the industry.

Thus, quantitative research on the application of innovation metrics will be a priority for further research. It is also interesting to use performance metrics for innovative projects.