Deriving evaluation indicators for knowledge transfer and dialogue processes in the context of climate research

. Knowledge transfer and dialogue processes in the ﬁeld of climate science have captured intensive attention in recent years as being an important part of research activities. Therefore, the demand and pressure to develop a set of indicators for the evaluation of different activities in this ﬁeld have increased, too. Research institutes are being asked more and more to build up structures in order to map these activities and, thus, are obliged to demonstrate the success of these efforts. This paper aims to serve as an input to stimulate further reﬂection on the ﬁeld of evaluation of knowledge transfer and dialogue processes in the context of climate sciences. The work performed in this paper is embedded in the efforts of the German Helmholtz Association in the research ﬁeld of earth and environment and is driven by the need to apply suitable indicators for knowledge transfer and dialogue processes in climate research center evaluations. We carry out a comparative analysis of three long-term activities and derive a set of indicators for measuring their output and outcome by balancing the wide diversity and range of activity contents as well as the different tools to realize them. The case examples are based on activities which are part of the regional Helmholtz Climate Initiative “Regional Climate Change” (REKLIM) and the Climate Ofﬁce for Polar Regions and Sea Level Rise at the Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research. Both institutional units have been working on a wide range of different knowledge transfer and dialogue processes since 2008/2009. We demonstrate that indicators for the evaluation must be based on the unique objectives of the individual activities and the framework they are embedded in (e


Introduction
Over the last few decades, the need for justification and legitimacy of the objectives and contents of publicly funded research in terms of quantitative and qualitative aspects has led to an increased use of evaluation approaches in almost all fields of research activities (e.g., in the field of climate adaptation, Arnott et al., 2016; in the field of co-producing climate science, Wall et al., 2017).The overall aim of these approaches is to determine whether particular activities within the research programs are tailored and work efficiently and successfully, and to demonstrate that resources are used in a responsible manner (e.g., Neij and Åstrand, 2006).This also applies to increasing efforts in the field of knowledge transfer and dialogue processes, which form an important part of science in modern societies and present a novel type of challenge for research centers.Even though new information and communication technologies have made it much easier for society to access research results, there is still a large gap between the knowledge produced by researchers and that used in practice (e.g., von Storch et al., 2015;Arnott et al., 2016).Regardless of whether discrepancies between the pro-R.Treffeisen et al.: Evaluation indicators for knowledge transfer vided and the perceived knowledge are institutional or societal in origin, knowledge transfer and dialogue processes between science and society can significantly contribute to reducing the gap between knowledge creation and knowledge use (e.g., Kirchhoff et al., 2013;Meinke, 2017).In the context of polar and marine research in the field of climate sciences and as part of the Helmholtz Association in the research field of earth and environment, the term knowledge transfer refers to the exchange of knowledge between science and members of society, with all groups of society being addressed (including representatives of politics, administration, business, civil society, education and media) (Hansjürgens et al., 2016).Knowledge transfer uses different formats, channels or instruments that meet the needs of the respective target groups and are suitable for the specific activity.It not only involves the provision of scientifically founded knowledge, but it also includes the translation and contextualization of knowledge, which is mostly target-specific.At best, knowledge transfer leads to the development and implementation of joint research projects (co-development) in which knowledge transfer turns into a collaborative approach of knowledge development.This tends to be more easily accepted by decision makers because they feel a stronger ownership of knowledge as it fits more likely to their own needs (Wall et al., 2017, and references herein).
Until now, however, it has been difficult to measure how successfully research centers engage in knowledge transfer and dialogue activities.This is mainly due to the lack of a common set of indicators and procedures to do so.Overall, few concepts are evidently very important in the establishment of indicators for evaluation processes, although external factors (e.g., changing policies and theme priorities) and impacts cause different initial conditions for transfer activities (e.g., Meinke, 2017;Wall et al., 2017).One concept which is widely applied, for example, by the European Commission in the " Development and Conceptual Results Framework" (EC, 2015), is based on the "logical model approach" (Julian et al., 1995;OECD, 2002;UNDG, 2010).It generally provides the opportunity to consider linkages between inputs, activities, outputs, outcomes (from immediate to long term) and potential impacts.This is one of the major strengths of the logical model as a planning and evaluation tool.As planning and evaluation go hand in hand, the logical model serves as a bridge between evaluation and activity planning.The logical model is a systematic and visual way to present and share the understanding of the relationships between the available resources to perform a knowledge transfer activity, the activity steps themselves, and the intended changes or results one hopes to achieve (see Table 1 for definitions) (e.g., Julian et al., 1995;Blotevogel and Wiegand, 2015).
In order to structure the activities discussed in this paper, we follow this logical model approach, which provides a linear sequence of steps that need to be taken for a project to meet its desired outcomes (e.g., Rush and Ogborne, 1991;Julian et al., 1995;Wholey et al., 2004).While for the imple-mentation of a project the determination of required inputs, such as human or financial resources, preconditions the activity, its overall result can be regarded in three different ways.At first, the direct output -for example, the product service or facilities that originate from the completion of the activity -provides a measurable result, which can be quantified and directly determined.Secondly, taking a closer look, the value for or effect on the considered stakeholder group can be considered, accordingly, which is described by the outcome of the activity.This requires a more qualitative view on the process and the respective effect.Finally, the long-term effect on the broader target group -for example, society -can be determined, representing the overall impact of the project.This, however, can only be quantified by additional accompanying research and is a complex and difficult evaluation field (e.g., Bornmann and Marx, 2013;Breteron et al., 2017).
Taking the special challenges of knowledge transfer and dialogue processes into account, only a handful of activities can be directly compared in terms of evaluation.Furthermore, one cannot equally measure all components in these processes as they normally differ in many aspects (e.g., objectives, boundary conditions or their setup; Meinke, 2017).Generally, it is important to keep the evaluation process focused, tailored and as simple as possible by means of clear indicators and measures.Otherwise, one runs the risk of overweighting the evaluation in terms of resources in relation to the activity itself (e.g., Crawford et al., 2017).Therefore, evaluation indicators should follow the SMART principle.This means that the indicators should be specific and measurable as well as achievable, relevant and time-bound (Doran, 1981).Furthermore, it is generally agreed that it is difficult to measure an activity's impact especially when the time period for data acquisition is too short (e.g., Pardoe, 2014;Brereton et al., 2017).Furthermore, it is also impossible to create an exhaustive list of indicators that can be directly applied to all knowledge transfer and dialogue processes in the context of activities performed by, for example, climate research centers.Hence, it is difficult to define success and its specific realizations in a generalized approach.Another important aspect with respect to evaluating knowledge transfer and dialogue processes is that these processes are not performed in isolation, outside or apart from their surrounding environments.Thus, they are largely affected by external factors and vice versa (e.g., Wall et al., 2017;Meinke, 2017).Among the elements that knowledge transfer and dialogue processes may affect are science-society interactions and relations (e.g., Kirchhoff et al., 2013, and references therein).All of these factors may have a major influence on the achievement of outputs and outcomes and, finally, on the impact.
This paper does not intend to provide a comprehensive and detailed account of different evaluation approaches nor an overview of proposed solutions to specific evaluation approaches for knowledge transfer and dialogue processes.Instead, it reviews three practical case examples in the field of Table 1.Definitions of components for the result chain of the knowledge transfer and dialogue processes in the context of evaluation processes (e.g., Julian et al., 1995;OECD 2004;UNDG, 2010) knowledge transfer activities: an information and data portal on the topic of sea ice (www.meereisportal.de)(Grosfeld et al., 2016b), the project "Science meets cinematic art" and the school project "Maritime Center Elbe Island (Maritimes Zentrum Elbinseln)" (see Table 2 for details).All three examples are based on the work of the Helmholtz Climate Initiative "Regional Climate Change" (REKLIM) and the "Climate Office for Polar Regions and Sea Level Rise".They are embedded in the overall research program structure of the Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research and, thus, are part of the future center evaluation in the field of knowledge transfer and dialogue processes.It needs to be mentioned here that the three case examples were originally launched without a comprehensive evaluation concept.The criteria and indicators for the evaluation of success and usability have been derived in retrospect and will be considered in the further course of the activities.The work presented here is also part of an ongoing discussion within the German Helmholtz Association regarding the evaluation of performed or ongoing activities in the field of knowledge transfer and dialogue processes.It should be noted that in contrast to Wall et al. (2017) the focus does not lie on co-production of knowledge, information use in decision-making or evaluating co-produced climate research but rather on a large range of formats, channels or instruments which are not solely co-produced climate research but arise from the wide understanding of knowledge transfer and dialogue processes of the German Helmholtz Association (Hansjürgens et al., 2016).In this framework the paper tends to derive a more generalized principle classification scheme of evaluation indicators based on our long-term experience with a large variety of activities and formats that allow an approach from a practitioner point of view (see Fig. 1).
The presented efforts intend to serve as a basis for upcoming research program evaluation and can provide assistance to evaluators by putting the achieved outputs and outcomes into the right conceptual context.The study of the case examples underlines the challenges that arise during the evaluation of activities, exhibiting a huge diversity in topics and activity contents and being embedded in an institutional frame.

Case examples in the field of knowledge transfer and dialogue processes
Climate research results may provide a more solid basis for climate-related decision support if the knowledge base for decision making and awareness can be improved through dialogue with specific actors (e.g., Spruijt et al., 2014;Vasileiadoua et al., 2011).A well-known example of such a process in climate science is the IPCC (Intergovernmental Panel on Climate Change) process and its summary for policy makers (IPCC, 2013) that laid the groundwork for the Paris Agreement in 2015.A second example is the Montreal Protocol on substances that deplete the ozone layer (Haas, 1991), which now, about 30 years later, shows as a result the first clear indications of a recovery of the Antarctic ozone layer (Solomon et al., 2016).
The case examples in the field of knowledge transfer and dialogue processes discussed originate from the work of the Helmholtz Climate Initiative REKLIM.REKLIM is a consortium of nine research centers of the Helmholtz Association that focuses on the investigation of representations and impacts of global climate change on a local to regional scale (Grosfeld et al., 2016a).Regional observations and process studies coupled with model simulations help to improve regional and global climate models.REKLIM contributes to the strengthening of multidisciplinary regional climate research in Germany and internationally.Together with the four regional Helmholtz Climate Offices (Schipper et al., 2009) the REKLIM Head Office (REKLIM-HO) develops tools and dialogue processes in order to transfer research results from the REKLIM initiative to the public and to other societal stakeholders.In return, dialogue process with stakeholders can reveal new research questions as well as foster and stimulate new research.
The transfer realized within the wide range of different performed transfer activities consists of all means by which knowledge may move during the knowledge transfer and dialogue processes (e.g., web platforms, discussion forums, brochures and print products) and allows the involved actors (i.e., researchers, knowledge brokers and practitioners) to enter a two-way exchange of knowledge and information (e.g., Hubermann, 1990;Lomas, 2007).Our work experience suggests that knowledge brokers are central actors in these processes.Here, both, REKLIM-HO and the regional Helmholtz Climate Offices operate as such.To provide a better overview of our activities, we grouped conducted and ongoing activities based on our experience in an idealized scheme in which the degree of stakeholder engagement of activities increases from left to right (Fig. 1).The groups are based on the one hand on a subjective segmentation and on the other hand on the long-term experience of the authors in the field.This exercise was performed to help sort the wide range of diverse knowledge transfer activities and dialogue processes broadly into groups.In this scheme we established three fields ranging from more general activities in segment I (summarized under the keyword "information") (e.g., Marín-Gonzílez et al., 2017) to highly specialized tailored activities (segment III; summarized under the keyword co-development).The latter emphasizes long-term perspectives and dialogic knowledge exchange with specific stakeholder groups (e.g., Ellen and Brown, 2016), which may eventually lead to sustainable impacts.Unfortunately, these cannot be evidenced easily.In our opinion, this idealized scheme serves to gain a general picture of the wide range of performed activities which are based on a variety of channels, formats or instruments.Still, all activities are based on the specific research foci of our research center where the knowledge brokers (REKLIM-HO and the regional Helmholtz Climate Offices) are set.
The first chosen activity is located in segment II.This segment encompasses activities which originate from our research foci and expertise but partly already take into account stakeholder engagement.We initiated the information and data portal on the topic of sea ice as the necessary scientific expertise is bundled at our research center (www.meereisportal.de;Grosfeld et al., 2016b).This activity was characterized by fewer dialogue components in the beginning, but stepwise the stakeholder involvement increases as we get more and more specific user requests.We now take a closer look at two activities located in segment III (Fig. 1).This group is concerned with our long-term activities, from which we expect the most probable impact on a long-term view.It is, on the one hand, the establishing of a specific school environment with a focus on marine themes in daily school life in a structurally and socially disadvantaged area in Hamburg, Germany (Maritime Center Elbe Island).On the other hand, we started an innovative media project where, among other aspects, science-related documentaries, viral videos and a blog covering an international scientific conference of REKLIM have been generated in cooperation with the DEKRA University of Applied Science for Media, Berlin

Derivation of indicators for knowledge transfer and dialogue processes
In the following section we discuss potential indicators for evaluating the output and outcome of each of our three case examples before we classify them into groups in the context of our framework.This attempt is driven by the complexity of the activities in the field of knowledge transfer and dialogue processes and the novelty to have them included in our climate research center evaluation.The specific framework in which the respective activities are performed, its conditions and objectives are taken into consideration.The general aim of determining evaluation indicators is to serve as an input to stimulate further reflection on the field of evaluation of knowledge transfer and dialogue processes in the context of climate sciences.Therefore these should be adjusted to each other now.The evaluation also helps to assess whether stakeholders broaden and improve their knowledge background based on established scientific knowledge and if the knowledge transfer activity prepares them for well-founded decision making.Some indicators serve as measures to quantify the measurable outputs/outcomes according to the logical model introduced above and to demonstrate whether a defined (activity) goal has been achieved.Indicators define the data to be collected and relate directly to the activity's key objectives.Quantitative indicators help to assess to what extent, for example, activities are on track and whether the anticipated outputs/outcomes have been achieved.Qualitative indicators record changes in the attitudes and behavior of the intended stakeholder group, which are often less tangible and, thus, difficult to quantify.They are generally more descriptive and often take different formats, such as narratives, pictures or stories (Costantino and Greene, 2003).Some indicators mainly rely on self-reporting.Furthermore, the indicators should be appropriate in the context of an activity's evaluation.
There is a lack of comparable definitions for indicators, and even their comparability is strongly limited, especially in the broad range of activities performed in the field of knowledge and transfer processes of research organizations (Meinke, 2017).Thus, when developing an evaluation system, the indicators have to be thoroughly defined, according to the particular frameworks and requirements.However, an evaluation of impacts arising from an activity often cannot be properly conducted by oneself as it requires costly supporting, accompanying socio-scientific expertise, which is rarely available in-house.In addition, the intended impacts mostly occur on a longer timescale and can only be observed through an in-depth and continuous analysis of change within a broader stakeholder community.All these challenges have to be addressed when establishing an indicator system.Furthermore, determining whether an activity can be considered successful or not depends on the respective definition of what constitutes "success" and on a baseline definition.In terms of measurability, outputs are typically tangible and more easily approached and measured objectively.The measurement of outcomes is often more difficult and typically done in a subjective manner due to rather "soft" indicators (e.g., Becker and Renger, 2017).Outcome indicators are less tangible and more conceptual results.These include the perception that project goals have been achieved and how end users perceive credibility, saliency and legitimacy of the final outputs and process (Wall et al., 2017).
In order to determine qualitative or quantitative indicators, associated measures have to be defined.These measuring tools can be simple numerical counting systems (e.g., no. of website visitors, no. of downloads, no. of prints of brochures or reports) or questionnaires that help to gain insights into the customers' perception of provided knowledge (e.g., W. K. Kellogg Foundation, 2004;Wall et al., 2017).Since all activities in the field of knowledge transfer and dialogue processes are very different and highly dependent on the target group's size and the transfer content, they cannot be quantitatively compared to each other.For example, an information portal on health issues is generally more relevant for society at large than a specific information tool on climate change.Nevertheless, depending on the target group the long-term development of information flow and reception of knowledge must be recorded to enable evaluation of an activity's output, outcome or impact with regard to its objective.In this case, the definition and determination of a baseline is important and helps to monitor the respective progress and long-term change.When deriving adequate evaluation indicators, one should also take into account that indicators may also have only limited conclusiveness.
Based upon these considerations and having intensively studied the literature, we identified a set of indicators in terms of output and outcome for our three different case examples following the SMART criteria (Table 3) (using, e.g., Doran, 1981;W. K. Kellogg Foundation, 2004;Wall et al., 2017).These indicators reflect the range of activities and also the various contents of these activities.Despite this diversity, the listed indicators show great overlap and numerous similarities.Thus, we established a principle classification scheme for the obtained indicators with the aim of sorting them and applying or transferring them beyond the three activities they are based on.For this purpose the different indicators were classified in terms of quantitative or more qualitative nature (first dimension) and whether they are more output-or outcome-related (second dimension).This leads to four superordinate groups with different characteristics.The amount of example indicators was kept to the minimum possible as they only represent a small selection of possible indicators of the respective group.The groups are designed to cover the breadth and variety of indictors (Fig. 2).
The two left-hand fields include indicators which can be represented mainly by numeric measures and deal with outputs whereby the indicators vary in their degree of validity and the nature of quantification.While the indicators in segment IV reflect the activity's general visibility, the field above (segment I) groups those indicators which transcend their quantitative character as they reflect a direct relation to the activity's context and content.This distinction emphasizes the fact that numeric measures may convey a different message.While the number of downloads on a website reflects the general usage and explains, for example, the visibility (e.g., indicators serving the criterion "dissemination/attention" for meereisportal.de in Table 2), the number of presentations at workshops or conferences at which a website is presented with its content and functionalities contains another type of information.In this case interested stakeholder groups are reached directly.Since the target group is smaller, it even allows a very direct and content-related information exchange and dialogue.This distinction aims to raise awareness for the almost qualitative nature of some quantitative indicators.Therefore, quantitative indicators should be handled carefully when they are compared.
The two right-hand segments present the outcome-related indicators.The lower segment III contains indicators that represent the activity's effect beyond its funding period (e.g., indicators serving the criterion "multiplying effect" in Table 2).They can be measured in terms of quantity (such as the amount of usage as a blueprint), but such a measurement should be accompanied by narrative components as their  qualitative nature is very high.These indicators reflect the activity's sustainability, among other things.Segment II in the upper right contains qualitative indicators which are more difficult to ascertain as they do not operate on numerical measures but rather present the activity's outcome -for example, in the form of descriptions, storytelling or qualitative scalesin the targeted stakeholder group (e.g., indicators serving the criterion "innovation character" in Table 2).However, qualitative indicators are very important for the evaluation process as they may present why certain activities develop in a specific way and, in the contexts of stakeholder's decisions, actions and perceptions related to an activity.According to Table 1, from these "precious" indicators eventually emerge the indicators related to the impact on society, which, however, is not discussed in this paper.The results from Fig. 2 can be very useful in the context of our research center evaluation in the field of knowledge transfer and dialogue processes for evaluators.

Conclusions
The combination of multi-disciplinary science, the increasing demand for up-to-date information and user-customized products emphasizes the need for new ways of communication between science and society.Research organizations are often confronted with obstacles when evaluating knowledge  et al., 2016).
In our experience, insights on the possible societal impact of knowledge transfer and dialogue processes can only be gained when taking the following aspects into account.First of all, activities require a certain degree of long-term orientation in order to guarantee a high level of confidence be-tween partners.This enables the successful development of joint "products" with one or more cooperation partners (codevelopment).However, the work in this field also highly depends on personal reputation and contacts.Furthermore, the activities' social level of attention and the perceived significance of their process, format and realization are determined by the particular theme.The diversity of involved actors and wide range of various objectives complicate the comparison of highly different activities and their relative significance in terms of influences on society.Generalized criteria of evaluation might not meet the different requirements and, consequently, undermine a critical and evaluative approach.Furthermore, the activities are largely based on developing processes, which result in a temporal gap between the realized effort and the intended reward for its results.A particular lesson we learned is the difficult distinction of qualitative and quantitative measures in terms of outcome.Overall, a valuable and successful impact can only be achieved through sustainable and trusted partnerships and long-term processes and solutions.

Figure 1 .
Figure 1.Idealized scheme for principle classification of working examples of REKLIM-HO and the Climate Office for Polar Regions and Sea Level Rise, according to the degree of stakeholder involvement in the specific activity as a distinctive feature.Highlighted (in bold) are the three case examples in the field of knowledge transfer and dialogue processes discussed in this paper.
. The selected three activities cover the following distinguished fields of knowledge transfer activities (Hansjürgens et al., 2016): information and consulting; knowledge exchange; and, finally, capacity building.Table 2 provides a brief description of these three case examples in terms of project aim, target group, responsible institution and possible evaluation criteria.It should be noted that in these case examples the evaluation criteria were not determined prior to project start.The three case examples form the basis for the analysis, which regards the determination of possible evaluation indicators of these activities in the field of knowledge transfer and dialogue processes in the frame of our research center evaluation.

Figure 2 .
Figure 2. Principle classification scheme of indicators derived from three case examples in the field of knowledge transfer of REKLIM-HO and the Climate Office for Polar Regions and Sea Level Rise.
transfer and dialogue processes as part of science in a modern society.This has to be taken into account when setting up an indicator system for, for example, research program evaluation purposes especially in an institutional context such as for the authors of this paper.This paper is based on longterm experience in the field of knowledge transfer and dialogue processes at a climate research center.It not only derives indicators based on three representative case examples but also merges them in an easy classification scheme.This result may support and enter into the general discussion of how indicators within a center evaluation of such activities can be used.The use of evaluation indicators must be carried out with great care.Therefore, this paper is very different to what, for example,Wall et al. (2017) present in their paper.WhileWall et al. (2017) put their focus on co-production of knowledge, information use in decision-making and evaluating co-produced climate research, our activities are based on a large range of formats, channels or instruments which are not solely co-produced climate research but arise from the wide understanding of knowledge transfer and dialogue processes of the German Helmholtz Association (Hansjürgens .

Table 2 .
Overview on the case examples in the field of knowledge transfer and dialogue processes.

Table 3 .
Overview on derived indicators for three working examples divided into output and outcome.The process followed the "logical model approach".