Pilot 6: Reflexivity of metrics on medical research & dissemination practices
The goal of the pilot study was to analyse how much the use of new dissemination practices and channels (such as blogs or social media) depends on established cultures of appreciation that reward usage of these channels.
To achieve this we engaged with a research community to analyse their use of social media tools and Altmetrics. Stakeholders of the communities were involved in interviews and meetings and asked to report their social media and Altmetrics activities. The goal was to create a community profile on Altmetrics and evaluate the applicability of the activity in this specific context. The evaluation covered the potential of the approach for informing about the suitability of specific metrics for the selected community.
For this pilot, we managed to cooperate with the Berlin Institute of Health (BIH). We analysed debates in editorials of major biomedical journals (N=144). Based on this analysis and the discussion we had with members of the BIH, we found that there may be different cultures of dealing with data in the biomedical sciences. Clinical, pre-clinical, and lab-oriented research have developed different practices and stances towards the handling, governing and acknowledging research. We therefore agreed to focus on data use and data stewardship at different stages of the biomedical enterprise (clinical and pre-clinical research). Clinical, pre-clinical, and lab-oriented research have developed different practices and stances towards the handling, governing and acknowledging research. On the basis of field work and exploratory analysis, we aim at providing recommendations for specific metrics appropriate to the needs of our community. Therefore, our goal was to identify barriers, enablers and constraints in Open Data in the biomedical research field. Based on this, we developed four different criteria for the evaluation of our pilot.
- The extent to which we identified problems related to the provision of open data in biomedicine
- Extent to which we identified community needs for the use of open data
- Extent to which we find field specific practices related to open data
- Extent to which we provide input of how to govern and incentivize the use and provision of open data in a biomedical research facility
In order to meet this goal, we developed a strategy which contained three elements. First, we assisted the process of monitoring the Open Data publication output in order to gain an overview about publication practices in the field and to select cases for exploratory case studies. Second, we aimed at constructing a search strategy which allows for identifying different ways of how biomedical researchers mention or link open data in publications. And, third, we aimed at conducting field studies at BIH research facilities both at the Charité and the MDC in order to explore current field specific data practices and potential institutional or social barriers for open data. Summarizing, our research design and our focus allowed us to carry out activities which are targeted and which respond to the needs of the community and which will put us in the position to derive balanced and field specific recommendations towards how novel forms of metrics can be established.
The preliminary results of the interviews revealed the aforementioned different data cultures in biomedicine there are in fact very different cultures in biomedicine regarding the handling and the provision of data. We found rather different accounts of problems, but also different solutions to the problem of disseminating data in biomedicine. The results of the interviews will be collected and discussed with the partners at BIH. Our main aim is to carve out the differences between the different fields, e.g. clinical research on the one hand and laboratory research (molecular biology) on the other. Referring to our questions, our main goal is to explore field specific problem perceptions and research practices which may hinder the provision of research. In addition, we also identified arguments of persons who accrue more critical stances. The main argument which has been repeatedly made is that, particularly in clinical research, the provision of data should not be promoted freely, but regulation should be clearer on who should share the data, for what purposes, and to what extent the data should be provided and shared with other researchers to prevent abuse. Currently, these problems only appear in clinical research, but it may be also relevant for laboratory research, as these more strongly orient on personalization of therapies.
Our pilot study has shown that Open Data is a relevant task for members of the biomedical research community. The results show that to answer these questions, one really has to engage with the members of the community. These are our main findings:
- The pilot has shown that providing incentives for Open Data provision is difficult. There are different regulations, guidelines which need to be addressed.
- Incentives for providing Open Data need to be field specific. Even in the biomedical realm, there is an enormous variety of different data cultures, that is, different stances, ways of reflecting, handling and valuing data which make a unified framework difficult. Thus, metrics for incentives should be field specific and reflect the respective epistemic practices.
- There is a lack of regulation regarding the governance of data usage. Particularly in the realm of clinical research, there is a need of a governance framework of who can access, who can use and alter the data, for what purpose, and at what point in time.
- To a certain degree, there is still a lack of institutional and organizational support, such as guidance and technical advice in Open Data principles, technologies, and practices. Mentorships, technical or infrastructural advice might benefit the provision of Open Data.
Read the full evaluation report here.