Pilot 4: Transferring the research life cycle to the web

The goal of the pilot study was to investigate whether qualitative research, in particular data analysis and data collection, can be transferred to open online groups. A special focus lied on mechanisms to engage citizens in qualitative research processes.

In a first step we further developed an existing OpenOnlineResearch (OOR) software prototype, which enables collaborative online data analysis and collection. In a second step, we tested the applicability of this online solution to involve citizens in qualitative research. The goal was to gain further insight into working practices and address current challenges/gaps of open online collaboration approaches applied to qualitative research.

Key Results

OOR builds on prototypes developed earlier at the University of Amsterdam. At that point, the prototypes were geared to academic participants who already had some experience with similar tools. In OpenUP we worked to make the software easier to use, which allowed to involve citizens that were not experienced in research or using the tool. We have performed three tests with team members and non-team members (uninitiated and untrained students from different disciplines plus non-academic and academic members of our network). In 2018, following further improvement of the tool, we had a software test with the extended team. In this test we both assessed the workings of the functionalities, the specific needs for (a minimum of) instructions and the CMS and output capacities so far. Five users went from log-in to interpretation to stacking repeatedly. All steps were analysed, and potential improvements were discussed. The test revealed no major flaws.

In September we have interactively demonstrated and tested the tool at the final OpenUP conference. We invited conference participants to use the interpretation features of OOR, visually observe the use of it, and asked them about usability, instructions and meaning of the interpretations given. Six participants have gone through the software and performed the main operations. It turned out that users understand the flow and meaning of the software swiftly and are able to produce meaningful content. It also turned out that we omitted a small number of instructions. The test also triggered participants to think about the applicability of the tool. For example: one participant suggested to try the tool for peer-review.

At the end of 2018 we will perform an ulterior test in Zooniverse. The test will focus on scaling up and the pitfalls we encounter when scaling up.

Pilot 4 demonstrated that open online interpretation of qualitative data is feasible and that yet unused parts of the research cycle can be opened to wider ranges of collaborators both within and outside academia. The results of the testing confirmed that the Open Online Research (OOR) tool enables online collaborative interpretation. We learned that a simple tool, without the need for detailed instruction, is feasible. We also learned that the input of scientists is still needed for the formulation of sound research questions and instructions. We have also seen that online collaboration needs moderation (either technically or by humans) to settle differences. However, we found that conflicts were rare and that participants were willing to collaborate in most cases.

The outcomes of this pilot are feeding into future developments in open science in two ways. First, the collaboration with Zooniverse is continuing and might lead to either an integration of OOR methodology in Zooniverse or to a strengthening of OOR. Second, the continuity is safeguarded by new funding appointed by the University of Amsterdam. Based on the progress made within OpenUP, the UvA was willing to invest into the development of service package for OOR. While OOR is designed to be open and freely available, certain users might have more elaborate needs when using the tool. The need for services is assessed with the funding. In short, OOR is evolving beyond OpenUP.

Read the full evaluation report here.

Contact Christian Bröer and Nynke Kruiderink, University of Amsterdam, C.Broer@uva.nl