The Selection Process for a Web-Based Application to Measure Patient-Reported Outcomes Following the Example of the TREAT NL Registry

TO THE EDITOR Research was conducted on behalf of the TREatment of Atopic eczema, the Netherlands (TREAT NL) registry to select a web-based application for the collection of patient-reported outcome measures (PROMs). The registry aims to collect long-term, real-life data from patients with atopic eczema on phototherapy and systemic immunomodulating therapy. The collected PROMs comply with the TREAT NL core dataset to ensure cross-country data uniformity (Vermeulen et al., 2019). We outline the requirements and steps in selecting the most suitable application. With this, we aim to guide researchers in selecting an application for a research-oriented collection of PROMs. The selection process starts with determining the must-have initial requirements by the investigators, who will be managers of the selected application. Examples of initial requirements for the TREAT NL registry are that the application has to provide or support the composition of the designated PROMs, the application has to work on mobile devices, and the costs of the application have to be feasible. The next step is to identify candidate applications on the basis of their availability. Applications that do not meet the initial requirements are then excluded. Next, detailed requirements need to be determined. We grouped detailed requirements into these categories on the basis of Volere (Robertson and

TO THE EDITOR Research was conducted on behalf of the TREatment of Atopic eczema, the Netherlands (TREAT NL) registry to select a web-based application for the collection of patient-reported outcome measures (PROMs). The registry aims to collect long-term, real-life data from patients with atopic eczema on phototherapy and systemic immunomodulating therapy. The collected PROMs comply with the TREAT NL core dataset to ensure cross-country data uniformity (Vermeulen et al., 2019).
We outline the requirements and steps in selecting the most suitable application. With this, we aim to guide researchers in selecting an application for a research-oriented collection of PROMs.
The selection process starts with determining the must-have initial requirements by the investigators, who will be managers of the selected application. Examples of initial requirements for the TREAT NL registry are that the application has to provide or support the composition of the designated PROMs, the application has to work on mobile devices, and the costs of the application have to be feasible.
The next step is to identify candidate applications on the basis of their availability. Applications that do not meet the initial requirements are then excluded.
Next, detailed requirements need to be determined. We grouped detailed requirements into these categories on the basis of Volere (Robertson and Robertson, 2012): investigator, legal, security, patient, feedback, and interoperability requirements. A literature search was conducted in PubMed, Google Scholar, and additional relevant sources to find information on conditions regarding legislation, security, patient preferences, feedback (i.e., of results to patients and clinicians), and interoperability (i.e., between systems).
Investigator requirements give insight into the preferences of the researchers who will manage the application.
Legal requirements must be met to comply with the General Data Protection Regulation, which applies to all organizations that process personal data in the European Union. Performing a privacy impact assessment before the data processing can be mandatory and may be required by the research institution (PrivazyPlan, 2018). National legislation may also apply.
The International Organization for Standardization provides guidelines for security requirements (ISO, 2016(ISO, , 2013a(ISO, , 2013b: NEN-7510, NEN-7512, and NEN-7513 (NEN, 2017, 2010. These standards must be met for security. The application should log activity for an audit trail and should have secure connections, with the data being only accessible to patients and researchers. An overview of potential patient requirements has been presented elsewhere (Snyder et al., 2009). Surveys should preferably be short and simple. Multiple questions can be presented per survey page, but too much page scrolling should be avoided. In addition, the ability to add free text in the survey is appreciated by patients. To facilitate data entry, mobile applications should be considered because these can be used at any time and on any device. The user experience of software needs to be assessed and optimized.
Feedback requirements relate to the presentation of results after the survey has been completed. Both patients and clinicians consider line graphs as the most useful and easiest to understand (Brundage et al., 2015). Displaying multiple time points is generally preferred by patients. Scaling should be uniform for all questionnaires. A scale from 0 to 100 is recommended (Brundage et al., 2015). Depending on the nature of the project, a link between the application and the electronic health record of participants can be considered. This may entail requirements to allow interoperability. The next step is to rank the importance of all the requirements in a template (Supplementary Table S1). This template is based on the work of Lawlis et al. (2001), Maiden and Ncube (1998), and Carnegie-Mellon University (2006). The requirements should be given a weight of importance from 1 to 10.
The candidate applications can be scored in the template thereafter. A score of 1 or 0 is given if the application, respectively, does or does not meet the requirement. A score of 0.5 is given when the requirement is met partly or will be met at a later point. Afterward, a weighted score is calculated for each requirement. The application with the highest total sum score is in theory the most suitable application.
The requirements described above are software requirements. There are also external factors to consider when choosing an application, for example, whether the application can be  Questionnaires can be modified within the application. 9 1 9 1 9 1 9 1 9 1 9 1 9 1 9 1 9 Questionnaires can be completed on mobile devices. Questionnaires can be sent to the patient automatically before the consultation or the patient can be automatically informed that new questionnaires are available.
10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 The patient can receive a reminder to fill out the questionnaires. 9 1 9 0 1 9 1 9 1 9 1 9 1 9 0 0 Application allows multiple types of inputs to answer questions (e.g., radio buttons, checkboxes, free text). Application allows questionnaires to be sent to family members of the patient and results registered in the application. 7 0.5 3.5 0.5 3.5 1 7 1 7 0.5 3.5 1 7 0 0 1 7 Data from the patient must be able to be removed by the healthcare provider.   1 9 1 9 1 9 1 9 1 9 1 9 1 9 1 9 If the system is unavailable, this is no longer than 5 min on a working day. Patients only have access to their own questionnaires and data. System logs activity for audit trail (NEN 7513). 9 1 9 1 9 1 9 1 9 1 9 1 9 1 9 1 9 Interoperability requirements Application can be linked to the webbased project database. Application is able to link to EHRs. 10 0 0 1 10 1 10 1 10 0.5 5 1 10 0.5 5 1 10 Application offers the possibilities to collaborate on (inter)national registration. Exchange of data or access to data is possible.  A limitation of the selection process for the TREAT NL registry was that patient requirements were not included in our template. However, patient requirements are of importance because patients are users of the application; hence, we included patient requirements in the selection process recommendations.
2 Specific requirements are the three subfeedback requirements.

3
Application allows data to be saved as SNOMED CT or LOINC code.
implemented within the institution, the amount of support or experience from the supplier, as well as the experience or preference within the institution. If the highest-scoring application is found to be unsuitable owing to one or more of these factors, the next application should be considered as the best alternative. The completed template guiding the selection process for the TREAT NL registry is provided in Table 1. Eight candidate applications were investigated for the TREAT NL registry. The selection process resulted in the choice for Castor EDC (Amsterdam, Netherlands), which scored among the best (percentage score of 90%). We learned that external factors, imposed by institutions, can be crucial in the decision process. An important decisive external factor for us was that at our institution, a connection was planned to be established between our electronic health record system and this application.
Testing and monitoring are advised before and directly after the implementation to assess whether the software works as intended and to ensure that users have an optimized user experience.
The proposed process to select an application uses the accompanying template. This provides a systematized overview of requirements that have to be considered and can provide a solid starting point for this process. We aimed for a process that is maximally manageable for researchers who do not have expertise in software engineering. Applying the process contributes toward optimizing users' expectancies and experiences on performance and effort, which influences the behavioral intention and use of the application (Koivumäki et al., 2008).
Learning from the process of selecting an application for the TREAT NL registry, the generalized and systematized selection process is presented to facilitate researchers worldwide in their decision making to find a suitable application to capture PROMs.

Data availability statement
No datasets were generated or analyzed during this study.