Index

Abstract

In response to the critical challenge of lack of interoperability amongst health information systems in South Africa, as highlighted by the National eHealth Strategy, the National Department of Health (NDoH) published the Health Normative Standards Framework (HNSF) for eHealth interoperability in 2014. The HNSF stipulated that all health information systems deployed in South Africa should conform fully to all applicable standards. The CSIR as a strategic partner to the NDoH, is developing a variety of health information systems that are deployed in public sector clinics and hospitals which need to be subjected to conformance tests. This research therefore aims to develop a process model that could consistently be used to test conformance of the various health information systems to the standards and profiles prescribed by the HNSF. The Goodhue Task-Technology Fit framework was adapted as the theoretical framework, since it holds that the probability of the process model having a positive impact on performance is increased, if the characteristics of the process model match the tasks that need to be performed. The Design Science Research (DSR) methodology and Action Design Research (ADR) process were used, since they operationalised the research towards a process model, to guide the process of testing eHealth systems for conformance to the standards prescribed in the HNSF. An important outcome of the research was that the ADR process was adopted for conformance testing at the CSIR and it will periodically be adapted as required to ensure its continued relevance and fit to the environment.

Keywords: Information systems, Standards, Norms, Healthcare.

JEL Classification: I19.

Received: 10 June 2020 / Revised: 15 July 2020 / Accepted: 18 August 2020/ Published: 9 September 2020

Contribution/ Originality

This study will address the critical challenge of lack of interoperability amongst health information systems in South Africa, by ensuring that they fully conform to all applicable standards. The conformance testing process model developed in this research study provides a mechanism for the Council for Scientific and Industrial Research (CSIR) in South Africa to continually test (at all stages of development) the conformance of its systems to the HNSF. In this manner, changes to the software design and code can be done during the development cycle, and not after systems have been deployed and are found wanting (when it is more difficult and more expensive to effect changes to the code).

1. INTRODUCTION

The World Health Organisation (WHO) and the International Telecommunications Union (ITU) define electronic Health (eHealth) as the use of information and communication technology (ICT) for health. In its broadest sense, eHealth is concerned with improving the flow of information, through electronic means, to support the delivery and management of health services (World Health Organization, 2012).

According to Terry (2014) the true value of eHealth in support of national health strategies is stifled by silo implementations and the resultant fragmentation of health information systems. An analysis of the health information systems deployed in South Africa revealed that there are at least 42 different systems deployed across South African health facilities (NDoH & CSIR, 2014). This study further reported that there is very little or no sharing of information across the various systems that have been deployed (NDoH & CSIR, 2014). The inability to share information across systems is the key challenge and this impact negatively on healthcare delivery in South Africa. Among the issues are the following:

Following the publication of the South African eHealth Strategy (National Planning Commission., 2012) efforts have been made to address the issue of fragmentation of eHealth systems. In 2014, the National Department of Health (NDoH) published the Health Normative Standards Framework (HNSF) for interoperability of eHealth systems. The HNSF strives to ensure that all South African eHealth systems conform to a minimum set of standards to enable information exchange and use (National Department of Health & Council for Scientific and Industrial Research, 2014). eHealth interoperability refers to the ability of health information systems to work together within and across organisational boundaries to advance the health status of, and the effective delivery of healthcare to, individuals and communities (HIMSS, 2013). The HNSF prescribes a minimum set of standards that all health information systems must conform to, in order to facilitate interoperability between diverse health information systems. In the preamble to the HNSF gazette, the South African Minister of Health stipulates that only those health information systems that fully comply with the standards prescribed in the HNSF, should be considered for procurement and implementation in the South African health system (National Department of Health & Council for Scientific and Industrial Research, 2014). At the time of conducting this study (2017/18), there existed no local capability to test whether health information systems did indeed conform to the standards prescribed in the HNSF. The absence of such a conformance testing capability meant that government and relevant regulators were unable to verify that health information systems do indeed meet the prescribed interoperability standards, and developers of health information systems are unable to continually verify that their systems meet the prescribed interoperability standards during the various stages of system development.

In light of the above, the aim of this study was to develop a process model that could describe and guide the process of testing the conformance of health information systems to the eHealth interoperability standards prescribed by the HNSF. This aim will be achieved by establishing the role that standards play in enabling eHealth interoperability; exploring best practice in conformance testing of health information systems to prescribed standards; and guiding an instantiation of a conformance test of a real-world health information system to one of the eHealth interoperability standards prescribed by the HNSF.

2. LITERATURE REVIEW

2.1. e Health

A systematic review of eHealth conducted by Oh, Rizo, Enkin, and Jadad (2005) yielded 51 unique definitions, and the study concluded that there is no consensus on the meaning of eHealth, but identified two universal themes associated with the term, namely, health and technology. At its 58th World Health Assembly, the World Health Organisation (WHO) described eHealth as “the cost effective and secure use of information and communications technologies in support of health and health-related fields, including healthcare services, health surveillance, health literature, and health education, knowledge and research (WHO, 2005). More recently, the WHO opted for a simpler definition of eHealth, referring to eHealth as the use of information and communications technologies for health (WHO., 2019). In this research, the simpler WHO definition of eHealth is adopted, due to the broad scope addressed by the definition, and hence its ability to encompass the fast growing range of applications of information and communications technologies in health.

In 2018, the WHO published the classification of eHealth interventions, aimed at categorising the different ways in which eHealth technologies are being used to support health system needs (WHO, 2018). Taking a summary of health system challenges as the point of departure, the classification of eHealth interventions introduces four categories, namely, interventions for clients, interventions for healthcare providers, interventions for health system or resource managers, and interventions for data services.

According to the national eHealth toolkit that was published by the WHO and ITU in 2013, the critical importance of these technologies for health has been known for a long time. However, health service provision encountered severe challenges in many countries before eHealth attracted sufficient attention to be moved from the periphery, to the centre of health planning (ITU, 2012).

The cost of acquiring the necessary hardware and software, as well as the cost of implementation and maintenance of eHealth applications, is one of the barriers to digital health adoption (Moseley, Randall, & Wiles, 2003). Measuring the return on such investment also proves to be difficult. McGee, Reeder, Regan, Kleinke, and Arnold (2009) argue that there is no strong business case for digital health, as the typical return on investment (ROI) is generally low and short-lived. Kiberu, Mars, and Scott (2017) identified three factors as hindrances to adopting telemedicine in Uganda, namely, lack of knowledge and skills, lack of policy, and resistance from healthcare workers. Resistance to change is another barrier to eHealth adoption. The implementation of eHealth requires changes to workflows and healthcare routines. Doctors and nurses may be threatened by such changes, as they can be perceived as questioning their professional judgement (Adebesin, Kotzé, Van Greunen, & Foster, 2013). Another reason behind resistance to eHealth that technology may compromise the personal relationship that they have with patients (McGrath, Arar, & Pugh, 2007).

While eHealth does have the potential to improve the quality of care, healthcare providers are concerned about the security of healthcare information (Anderson, 2007). Healthcare providers may be held legally liable for unauthorised access to healthcare information (Boonstra & Broekhuis, 2010).

In South Africa, there is recognition that the various health policy imperatives (including the National Health Insurance) will only be possible if they are supported by appropriate underlying health information systems (National Planning Commission, 2012); (National Department of Health, 2012).

2.2. eHealth Interoperability

There is no single definition of the term interoperability and the most popular one is provided by the Institute of Electrical and Electronics Engineers (IEEE), The IEEE describes interoperability as the ability of two or more systems or components to exchange information and to use the information that has been exchanged (Geraci et al., 1991). Vernadat (2007) describes interoperability as the “ability of a system (or process) to use information and/or functionality of another system (or process) by adhering to common standards”. Interoperability in eHealth also needs to address policy and organisational issues that reflect the main purpose of eHealth, namely, providing better, safer and more efficient healthcare delivery (NHTA, 2007).

Interoperability of health information systems has the potential to benefit all stakeholders involved in the delivery and receipt of healthcare (HIQA, 2013) which includes enhanced quality and safety of treatments received, electronic transfer of prescriptions, efficiencies through a reduction in duplication of data capture,  cost savings from the reduction induplicate diagnostic testing, etc. (Iroju, Soriyan, Gambo, & Olaleke, 2013).
eHealth interoperability is not a simple task, since the healthcare domain is a complex one and comprises a number of different stakeholders and actors, including doctors, nurses, radiologists, pharmacists, psychiatrists, medical aids, and others. Each of these actors generate the data that they require, which data is also complex, because it ranges from patient administration data to clinical data, billing information, and laboratory data (Iroju et al., 2013).

The prevalence of healthcare legacy systems with limited interoperability and compliance to eHealth standards is also a significant barrier to achieving interoperability. In South Africa, 42 different eHealth systems were identified, with very little compliance to standards and interoperability (National Department of Health & Council for Scientific and Industrial Research, 2014).

Some vendors use a lack of interoperability to their advantage as a customer retention strategy by building systems that can only interoperate with their own products (HIQA, 2013). The healthcare industry is traditionally paper based, and the transition to electronic systems present a significant change – one that is often met with much resistance (Iroju et al., 2013). Although the benefits of interoperability in healthcare are considerable, they may not be clearly visible as the benefits are dispersed across a large number of stakeholders such as vendors, providers, policy makers, and the individual (HIQA, 2013).

2.3. eHealth Interoperability Frameworks

In 2012, the European Commission set the objective to develop an eHealth European Interoperability Framework in the context of the generic European Interoperability Framework (EIF) (eHealth Network, 2017). Six principles were extracted from the generic European Interoperability Framework (EIF), namely, security and privacy, transparency, preservation of information, reusability, technological neutrality and adaptability, and openness (eHealth Network, 2017). For each interoperability level, the organisations involved should formalise cooperation arrangements in interoperability agreements. This plays a crucial role in the context of the eHealth EIF, as an interoperability agreement is an essential tool to accelerate the transformation process to achieve higher degrees of eHealth interoperability. In terms of legal interoperability, eight binding instruments and six non-binding legal instruments are pertinent to the work of the eHealth EIF.

From an organisational perspective, three recommendations have been extracted by the eHealth Governance Initiative (eHGI) discussion paper on semantic and technical interoperability to support the development of the eHealth EIF. These included encouraging greater cooperation between Member States; between national authorities and standardisation bodies; and  incentivisation of healthcare providers (eHealth Governance Initiative, 2012). With respect to semantic interoperability, three categories of semantic artefacts are proposed, namely, a systems for concept representation; clinical models that assemble data items and map each item to specific terminology subsets, and EHR information models that provide a higher level containment framework and provenance context (eHealth Governance Initiative, 2012).

The Refined eHealth European Interoperability Framework changed the eEIF from four to six levels. The Organisational level is split into Policy making and Care execution, because these levels require different actors and responsibilities. The governance of the collaboration is also anchored at the policy level, but affects all levels. The Technical level is split into Applications (i.e., health-specific technology), and IT infrastructure (i.e., general technology, servers, networks, etc.), because these levels have different responsibilities and obey different classes of standards.

The Australian National eHealth Transition Authority (NEHTA) published Version 2 of the NEHTA Interoperability Framework (IF2) in 2007. The purpose of IF2 Figure 1 is to serve as a “common reference point that provides guidance to business and IT experts in delivering interoperable eHealth systems in Australia – while allowing for the evolutionary and emergent aspects of business, policy and technology (NHTA, 2007).
Figure 1 NEHTA Interoperability Framework

Figure-1. NEHTA Interoperability Framework V2.

Source: NHTA. (2007)

2.4. eHealth Standardisation 

According to the International Organisation for Standardisation (ISO), a standard is “a document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context (ISO, 2004). It is important to note that consensus does not imply unanimity, but refers to “general agreement, characterised by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments (Benson & Grieve, 2016).

In the eHealth context, standards refer to the specifications that enable interoperability among healthcare-related information and communication technologies and systems that are typically developed by a multitude of different system developers. Standards are not software or hardware but, the blueprints that technology developers use to create products that will inherently be compatible with other products adhering to these same standards (ITU, 2012). Standards therefore act as the middle ground where coordination between different software systems is needed. For example, systems that have very different user interfaces can still communicate meaningful data if they use similar terminologies, or terminologies that can be mapped to each other, to capture information (Hawn, 2009; ITU, 2012). Standardisation offers a number of benefits, including the prevention of single vendor lock-in, promotion of healthy market competition with associated cost savings, reduction in the risks of new technology development, and removing the need for expensive customized solutions (Kotze, Foster, Van Greunen, & Adebesin, 2013; Wager, Lee, & Glaser, 2017). The benefits of standardisation increase at an exponential rate as the number of systems that need to be linked increases (Benson & Grieve, 2016). The number of distinct links required to interconnect N different systems increases according to the following formula: N(N-1)/2. Therefore, linking two systems requires only one link, while linking 6 systems requires 15 links, and linking 100 systems requires 4950 interfaces. Standards allow for the replacement of individual links between systems by standard-based interfaces, as described in Figure 2 thereby drastically reducing the effort involved in linking large numbers of nodes or systems.

Figure-2. Point-to-point links vs. Standards-based interoperability.

Source: Benson and Grieve (2016).

The literature review was comprehensive in that it firmly established the need for eHealth interoperability, and the critical role that adherence to standards plays in enabling eHealth interoperability. It further explored the field of testing for conformance to standards. The literature study therefore established a scientific grounding on which the aim and objectives of this research study could be met.

3. RESEARCH METHODOLOGY1

The conformance testing process model was primarily used by CSIR software developers and testers who were involved in the development of health information systems, since it is was envisaged that the conformance testing process model will also be of specific interest to the National Department of Health (NDoH)towards enabling national conformance testing of health systems. A process model is described by Nilsen (2015) as the specification of steps, stages, or phases in a process, in order to describe and guide the process of translating research into practice and as such, it extends to the implementation and use of research within a specific context.

The researcher adapted the Task-Technology Fit (TTF) as the theoretical framework, since the TTF theory proposes that the probability of information technology (IT) having a positive impact on intended performance is increased, if the capabilities of the IT artefact matches the tasks that needs to be performed (Goodhue, 1995; Goodhue & Thompson, 1995). These researchers assert that “[I]nformation systems (systems, policies, IS staff, etc.) have a positive impact on performance only when there is correspondence between their functionality and the task requirements of the user (Goodhue & Thompson, 1995). As depicted in Figure 3 the scope of TTF can be seen as the fit of Information system functionality to task requirements.

Figure-3. Task-technology fit.

Source: Goodhue and Thompson (1995).

The researcher adopted the TTF Figure 3 model to evaluate the fit of the conceptualised Process Model to Test Conformance of Health Information Systems to eHealth Interoperability Standards to the task of testing Health Information Systems’ conformance to eHealth Interoperability Standards.  The TTF model Figure 3 presumes that performance is impacted by the fit between three constructs: technology characteristics, task requirements, and individual abilities. This research argues that a fit exists between the process model characteristics and task characteristics within a localised context. This is represented as the Task Process Fit model (TPF) depicted in Figure 4 below. The localised context is framed by the South African Health System characteristics in general and the CSIR domain specifically, and is represented by Context Characteristics.

Figure-4. Task-process model fit.

Source: Goodhue and Thompson (1995).

With respect to this study, context characteristics are the local, CSIR, and national health systems in South Africa, and circumstances that form the setting for conformance testing. Task-Process Model fit depicted in Figure 4 is seen as the extent to which the Process Model to Test Conformance of Health Information Systems to eHealth Interoperability Standards, conceptualised in this research, could affect Conformance Testing of Health Information Systems to eHealth Interoperability Standards. Performance impact is described by Goodhue and Thompson (1995) as the “impact in this context related to the accomplishment of a portfolio of tasks. Higher performance implies some mix of improved efficiency, improved effectiveness, and/or higher quality.” Utilisation is described as “the behaviour of employing the technology in completing the task (Goodhue & Thompson, 1995).

4. RESEARCH APPROACH

An iterative design process, facilitated by a Design Science Research (DSR) methodology, was followed to explore and describe a process model as a research artefact, and to evaluate it through an instantiation (Hevner, March, Park, & Ram, 2004; Hevner & Wickramasinghe, 2018). The instantiation provides a real-world concretisation of the process model towards validation of the artefact (Gregor, 2006). Design Science Research complements the pragmatic perspective in the design, application, and evaluation of designed artefacts. Simon (1996) argues for the knowledge underlying the construction of artefacts as theory and describes the design process as having a satisfactory, rather than an optimum, design. Hevner et al. (2004) echo these sentiments and add the criterion that the solution 'does work', and the need to describe an environment in which it works. Gregor and Hevner (2013) additionally advocate the inclusion of design criteria in addition to a design artefact, arguing that “both the contributions made in the form of viable artefacts and the contributions at more abstract levels” constitute a DSR contribution.

DSR thus involves the construction of a wide range of socio-technical artefacts, such as systems, models, strategies, methods, and interventions (Hevner & Chatterjee, 2010). For the purpose of this study, the DSR method was operationalised through the Mullarkey and Hevner (2019) elaborated Action Design Research (ADR) process model. The ADR process model is considered appropriate as it caters for an immersive exploration as part of an in-situ context (Mullarkey & Hevner, 2019). This research is considered to have a problem-centred entry point, as it is driven by the researcher-practitioner's desire to understand what a “better artefact would look like (Peffers, Tuunanen, Rothenberger, & Chatterjee, 2007). As such, the ADR process model from DSR is adopted, since its iterative and in-situ nature is in line with the research aim, namely to develop a process model that can describe and guide the process of testing conformance of health information systems to the eHealth interoperability standards prescribed by the HNSF.

A number of research processes have been identified to operationalise the three-cycle model of Hevner et al. (2004) which evolved to a four-cycle model by including an additional cycle (Drechsler & Hevner, 2016). The ADR process depicted in Figure 5. Mullarkey and Hevner (2019) is adopted as it is iterative and in-situ nature is in line with the study's aim and context.

Figure-5. The four stages of action design research.

Source: Mullarkey and Hevner (2019).

Within the four stages of the ADR process, each states the objective of that stage, and accommodates the creation of an artefact, its evaluation, reflection, and formalisation of learning (Mullarkey & Hevner, 2019). There are various entry stages to the process and since the research aims to develop a process model that can describe and guide the process of testing conformance of health information systems to the eHealth interoperability standards described in the HNSF, this study utilises a problem-centred ADR entry point as it initiates the research by considering the problem.

The diagnosis cycle is described as facilitating the exploration and description of the application domain. The narrative in this cycle aims to establish the context within which this research study takes place. It firstly defines eHealth, and discusses its application, benefits, and barriers to adoption. It also provides a summary of some of the pertinent eHealth initiatives in South Africa and introduces the concept of eHealth interoperability. After defining eHealth interoperability, the importance and barriers to interoperability is given, and a summary is provided of some of the existing international eHealth interoperability frameworks.

The critical importance of standards as a mechanism to facilitate eHealth interoperability is presented. Describing standards in the eHealth context, the cycle argues the benefits of standardisation in addition to some of the challenges related to eHealth standardisation. Pertinent international standards development organisations and standards initiatives in South Africa, related to eHealth, are presented.

The design phase enables the development and evaluation of design features; various design principles are formulated as part of the design. In agreement with Sein, Henfridsson, Purao, Rossi, and Lindgren (2011) the design cycle in this research study is implicitly executed and explicitly presented (Mullarkey & Hevner, 2019) as the conceptual process model to test conformance of health information systems to eHealth interoperability standards.

In the implementation cycle, the conceptual process model for conformance testing was used to test conformance of one of the health information systems developed by the CSIR to the Patient Demographic Query profile prescribed in the HNSF. This real-life instantiation provided the opportunity to evaluate the conformance testing process model. In the process model of Peffers, Tuunanen, Rothenberger, and Chatterjee (2008) the activities are described that are associated with the demonstration of an instantiation as involving the substantiation of the efficacy of the produced artefact. This extends to include the artefact's use in experimentation, simulation, and similar activities. The demonstration of the artefact's worth implies an extensive knowledge of how the artefact is envisaged to solve the identified research problem. Hevner et al. (2004) describe a number of the available design evaluation methods. These, along with the specific methods applied to the research study, are presented in Table 1.

Table-1. Design evaluation methods.

Design evaluation methods Methods applied in this study
1. Observational Case study, field study  
2. Analytical Statistical analysis, architecture analysis, optimisation, dynamic analysis In this study, an analytical analysis was conducted on the domain expert reviewer’s feedback questionnaires.
3. Experimental Controlled experiment, simulation  
4. Testing Functional testing, structural testing  
5. Descriptive Informed argument, scenarios A descriptive analysis was demonstrated through an instantiation and critical reflection of the secondary documents and reports generated.

Source: Hevner et al. (2004) .

In this research, the method of domain expert review was selected for evaluation of the conformance testing process model. Expert review is described as the evaluation of an artefact, by a domain expert, against a set of evaluation criteria (Kovesdi & Joe, 2017). It is envisaged to provide authoritative and independent reviews on the validity of the proposed solution in action. A domain expert can be defined as an individual with specific relevant skills and extensive experience and knowledge in a specific area of expertise, according to IGI Global. (2015). MacLellan. and Soden (2003) and Glaser and Chi (1988) in deliberating the notion of an expert, argue that an expert can be viewed as an individual that is able to ruminate on phenomena within their domain effectively and strategically, by utilising their domain knowledge and experience. This outline implies sufficient insight and experience in a particular domain to enable the individual to make informed decisions related to a phenomena, thus enabling a critical evaluation (Holbrook, Krosnick, Moore, & Tourangeau, 2007; Jansen & Hak, 2005). It follows that a domain expert can be considered as possessing extensive knowledge in a specific field.

Domain experts specifically in the field of health information systems integration were selected who had specialised knowledge and experience of health standards and the role thereof in conformance testing. Holbrook et al. (2007) and Nielsen (2000) provide compelling arguments for the conservative selection of two (2) to five (5) experts as being sufficient, with feedback reaching saturation. For this study, five domain experts were used to perform the expert review. The domain expert review was conducted in two phases – ex-ante and ex-post. The ex-ante review consisted of a guided walkthrough and explanation of how the process was conceptualised and how it is envisaged to be instantiated. Suggestions were then incorporated where relevant. The ex-post domain expert review likewise consisted of a guided walkthrough of the instantiation and resulting process model. The steps recommended by Pinelle and Gutwin (2002) were followed as a guided walkthrough with the selected experts.

Hevner et al. (2004) state that a researcher might iterate back to the design and development activity to refine the solution artefact. Peffers et al. (2007) argue that, during artefact evaluation, the researcher observes and measures whether the research artefact provides a suitable solution to the research problem. He suggests that the objectives of the solution be matched with the practical outcomes of the artefact instantiation. They furthermore note that the evaluation entails an estimation of the fit concerning the functionality provided by the research artefact and with objective performance measures, as described in the conceptual artefact. The evolution cycle describes the re-consideration of the instantiated artefact after implementation and during adoption. 

5. RESEARCH FINDINGS

The findings are summarised below by linking to the specific research questions.

Question 1: What is the significance of standards in enabling eHealth interoperability?

Standards are a critical building block towards eHealth interoperability, as they provide a common ‘language’ that various health information systems can use to exchange electronic information in such a manner that exchanged information can be ‘understood’ and therefore used in a meaningful way (Dixon, 2016). In the absence of standards, interoperability between systems can be established by creating point-to-point linkages between the relevant systems. This mechanism of establishing interoperability is susceptible to the phenomenon of combinatorial explosion; while only one link is required to connect 2 systems, 15 links are needed to connect six systems, and 4950 interfaces are required to connect 100 systems (Oemig & Snelick, 2016). Standards offer an alternative to establishing interoperability by standardising the actual format and content of the messages that are being exchanged. In this manner, messages can be exchanged and ‘understood’ by various systems without the need for proprietary point-to-point links.

While the need for standardisation is widely appreciated, the selection of appropriate standards for eHealth interoperability is not a trivial task. This is because there are a number of international standards development organisations that are developing standards that do not necessary complement each other. In some instances, there are standards that actually contradict each other. Adebesin (2014) developed a method for the selection of eHealth standards to support interoperability in healthcare information systems. This method was used to develop the South African Health Normative Standards Framework for eHealth Interoperability (HNSF). The HNSF continues to be a valuable ‘policy’ that prescribes the minimum set of standards to which eHealth systems in South Africa must conform.

Question 2: What should the role of conformance testing be in enabling eHealth interoperability?
This study has firmly established the critical role of standards in enabling eHealth interoperability, and emphasised the HNSF as the foundation ‘policy’ that prescribes the minimum standards to which health information systems in South Africa must conform. The mechanism to establish whether such systems do indeed conform to the prescribed standards is conformance testing. Conformance testing assesses the extent to which systems meet the requirements that are inherent in the prescribed standards (Kindrick, Sauter, & Matthews, 1996).

The absence of conformance testing for the verification of conformance to standards means that developers and procurers of health information systems cannot be entirely sure that the health information system that they are building or buying conforms fully to the required standards. The cost associated with changing systems that are already deployed to make them standards compliant can be quite exorbitant.

Conformance testing cannot be substituted by interoperability testing, which often takes the form of connectathons, as these tests merely test whether systems can interoperate or not. They are unable to pronounce on whether interoperability or lack thereof is due to conformance on non-conformance of individual systems to the requirements of the standard. Conformance testing, for both developers and procurers of health information systems, should therefore be compulsory.

Question 3: What are the key elements that guide conformance testing of eHealth systems?
Conformance testing assesses whether a health information system meets the requirements of the underlying standard or not (Kindrick et al., 1996). The system requirements are derived from the underlying standards applicable to the functionality of that system. Therefore, before any form of conformance testing takes place, there must be clarity with respect to the underlying standards and specifications required by the SUT.
The conformance test plan should include test cases that test the SUT’s conformance to each of the requirements of the underlying standards. In other words, there should be a good match between the requirements of the standards and the proposed test cases. Conformance testing can be done manually, or by using automated test tools, or by a combination of manual and automated processes. There are a number of existing automated conformance test tools; some of these are openly available, and can be used to perform common conformance tests.

Configuration of the SUT and the chosen test tool/s can be a complicated task. It is vitally important that such configuration is well documented for future reference.
Main Research Question: What process can consistently be used to test conformance of health information systems to eHealth interoperability standards as described in the HNSF?
This study resulted in the development of an eight-step process model for conformance testing of health information systems, as summarised in Figure 6. Blocks in red indicate steps that were included as a result of the expert review evaluation process.

Figure-6. Eight-step model for conformance testing of the health operations information systems.

5.1. Significance and Contribution of the Research Study

In the preamble to the National eHealth Strategy the Minister of Health, Dr Aaron Motsoaledi, highlighted the fragmentation of health information systems as a major obstacle to the realisation of the benefits of eHealth in South Africa. The HNSF is generally considered a major step forward in addressing the problem of fragmentation of health information systems, as it prescribes the minimum standards that must be adhered to in order to enable interoperability of health information systems. However, the impact of the HNSF in terms of enabling eHealth interoperability is impeded by the absence of a mechanism to test whether health information systems are indeed conforming to the standards prescribed in the HNSF or not.

The absence of such a conformance testing mechanism leaves developers of health information systems unsure of the level of conformance of these systems to the HNSF. It is imperative that these systems interoperate with other facility-based and national health information systems. The CSIR therefore has a need to ensure that the systems it develops conform fully to all of the relevant standards prescribed in the HNSF. The conformance testing process model developed in this research study provides a mechanism for CSIR to continually test (at all stages of development) the conformance of its systems to the HNSF. In this manner, changes to the software design and code can be done during the development cycle, and not after systems have been deployed and are found wanting (when it is more difficult and more expensive to effect changes to the code).

The National Department of Health & Council for Scientific and Industrial Research (2014) calls for all purchasers of health information systems to consider only systems that are fully compliant with the standards prescribed in the HNSF. Since the publication of the HNSF, tender documentation from various purchasers have included HNSF compliance as a requirement. However, the absence of a mechanism to test conformance to the standards prescribed in the HNSF means that purchasers cannot be absolutely sure of the claims of compliance made by health information system vendors. The conformance testing process model provides an avenue for purchasers of health information systems to insist on systems testing and the provision of evidence of compliance.

5.2. Scope and Limitations of the Research

The conformance testing process model developed in this study ends with the production of test results that indicate whether the SUT has passed the specific conformance test or not. It does not produce an official certificate of conformance. In order for the process to include certification, it must be conducted in a laboratory that is ISO 17025 compliant. The conformance testing process model is limited by existing test tools and should the tools for specific tests not exist, then the test tool needs to be developed first, before the conformance testing process model can be employed.

The conformance testing process model tests conformance to the standards prescribed in the HNSF, which is not a comprehensive list of eHealth standards. It focuses only on eHealth applications that deal with patient information (National Department of Health & Council for Scientific and Industrial Research, 2014). Therefore, the conformance testing process model inherits the limitations associated with the HNSF.

The process model for conformance testing developed through this study was limited to use by the CSIR to test conformance of the health information systems it develops to applicable eHealth interoperability standards. At the time of the study, the HNSF focussed on interoperability standards for the exchange of patient-centric information, such as patient demographic information, summaries of care, discharge summaries, referral notes, and diagnostic test results. While also important for eHealth interoperability, the HNSF does not extend to specifying standards for edge devices such as computers, mobile devices, or portable diagnostic devices.

While Government and industry would likely be interested in the adoption of the conformance testing process model, this study focussed primarily on the application of the conformance testing process model in the CSIR environment – to test health information systems that are being developed by CSIR for conformance to the eHealth interoperability standards prescribed by the HNSF.

This study focussed on conformance testing, that is, ascertaining the degree to which an implementation of a standard meets the requirements of that standard (Moseley et al., 2003). The study specifically excluded interoperability testing, which is concerned with whether multiple systems can actually interoperate in real world environments. Interoperability testing can be defined as “the assessment of a product to determine if it behaves as expected while interoperating with another product (Kindrick et al., 1996).

Funding: This study received no specific financial support.  

Competing Interests: The authors declare that they have no competing interests.

Acknowledgement: All authors contributed equally to the conception and design of the study.

REFERENCES

Adebesin, F., Kotzé, P., Van Greunen, D., & Foster, R. (2013). Barriers & challenges to the adoption of e-health standards in Africa. Port Elizabeth: Health Informatics South Africa (HISA).

Adebesin, T. F. (2014). A method for the selection of e-health standards to support interoperability in healthcare information systems. (PhD (IT)), Nelson Mandela Metropolitan University, Pretoria, South Africa.  

Anderson, J. G. (2007). Social, ethical and legal barriers to e-health. International Journal of Medical Informatics, 76(5-6), 480-483.

Benson, T., & Grieve, G. (2016). Principles of health interoperability: SNOMED CT, HL7 and FHIR: Springer.

Boerma, J. T., & Stansfield, S. K. (2007). Health statistics now: Are we making the right investments? The Lancet, 369(9563), 779-786. Available at: https://doi.org/10.1016/s0140-6736(07)60364-x.

Boonstra, A., & Broekhuis, M. (2010). Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research, 10(1), 231. Available at: https://doi.org/10.1186/1472-6963-10-231.

Dixon, B. (2016). Health information exchange: Navigating and managing a network of health information systems: Academic Press.

Drechsler, A., & Hevner, A. (2016). A four-cycle model of IS design science research: Capturing the dynamic nature of IS artifact design. Paper presented at the 11th International Conference on Design Science Research in Information Systems and Technology (DESRIST) 2016. St. John, Canada, 23-25 May.

eHealth Governance Initiative. (2012). Discussion paper on semantic and technical interoperability. Retrieved from: www.ehgi.eu .

eHealth Network. (2017). The new European interoperability framework homepage. Retrieved from: https://ec.europa.eu/isa2/eif_en.

Geraci, A., Katki, F., McMonegal, L., Meyer, B., Lane, J., Wilson, P., . . . Springsteel, F. (1991). IEEE standard computer dictionary: Compilation of IEEE standard computer glossaries in IEEE Std 610 (pp. 1-217): IEEE Press.

Glaser, R., & Chi, M. T. (1988). Overview. The Nature of Expertise the Nature of Expertise, 15-18.

Goodhue, D. L. (1995). Understanding user evaluations of information systems. Management Science, 41(12), 1827-1844.

Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual performance. MIS Quarterly, 6(1), 213-236.

Gregor, S. (2006). The nature of theory in information systems. Management Information Systems Quarterly, 30(3), 611 - 642. Available at: https://doi.org/10.2307/25148742.

Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. Management Information Systems Quarterly, 37(2), 337-355. Available at: https://doi.org/10.25300/misq/2013/37.2.01.

Hawn, C. (2009). Take two aspirin and tweet me in the morning: How twitter, facebook, and other social media are reshaping health care. Health Affairs, 28(2), 361-368.

Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. Management Information Systems Quarterly, 28(1), 75-105.

Hevner, A. R., & Wickramasinghe, N. (2018). Design science research opportunities in health care. Theories to Inform Superior Health Informatics Research and Practice: Springer.

Hevner, A., & Chatterjee, S. (2010). Design research in information systems. New York: Springer.

HIMSS. (2013). HIMSS dictionary of healthcare information technology terms, Acronyms and organisations. Chicago: Healthcare Information and Management Systems Society.

HIQA. (2013). Health information and quality authority: Overview of healthcare interoperability standards. Health Information and Quality Authority. Retrieved from: https://www.hiqa.ie/sites/default/files/2017-01/Healthcare-Interoperability-Standards.pdf .

Holbrook, A. L., Krosnick, J. A., Moore, D., & Tourangeau, R. (2007). Response order effects in dichotomous categorical questions presented orally: The impact of question and respondent attributes. Public Opinion Quarterly, 71(3), 325-348. Available at: https://doi.org/10.1093/poq/nfm024.

IGI Global. (2015). What is a domain expert. [Online]. Available: [Accessed 2018/11/06].

Iroju, O., Soriyan, A., Gambo, I., & Olaleke, J. (2013). Interoperability in healthcare: Benefits, challenges and resolutions. International Journal of Innovation and Applied Studies, 3(1), 262-270.

ISO. (2004). Guide 2–standardization and related activities-general vocabulary. International Standards Organisation. Retrieved from: https://www.iso.org/home.html .

ITU. (2012). E-health standards and interoperability. ITU-T Technology watch report. Retrieved from: https://www.itu.int/dms_pub/itu-t/oth/23/01/T23010000170001PDFE.pdf .

Jansen, H., & Hak, T. (2005). The productivity of the three-step test-interview (TSTI) compared to an expert review of a self-administered questionnaire on alcohol consumption. Journal of Official Statistics: An International Quarterly, 21(1), 103-120.

Kiberu, V. M., Mars, M., & Scott, R. E. (2017). Barriers and opportunities to implementation of sustainable e-health programmes in Uganda: A literature review. African Journal of Primary Health care & Family Medicine, 9(1), 1-10. Available at: https://doi.org/10.4102/phcfm.v9i1.1277.

Kindrick., J. D., Sauter, J. A., & Matthews, R. S. (1996). Improving conformance and interoperability testing. Standard View, 4(1), 61-68. Available at: https://doi.org/10.1145/230871.230883.

Kotze, P., Foster, R., Van Greunen, D., & Adebesin, F. (2013). A Review of interoperability standards in e-health and imperatives for their adoption in Africa. South African Computer Journal, 50(1), 55-72.

Kovesdi, C., & Joe, J. (2017). A novel tool for improving the data collection process during control room modernization human-system interface testing and evaluation activities (No. INL/CON-16-40406. Idaho National Lab.(INL), Idaho Falls, ID United States.

MacLellan, E., & Soden, R. (2003). Expertise, expert teaching and experienced teachers’ knowledge of learning theory. Scottish Educational Review, 35(2), 110-120.

McGee, N. M., Reeder, G., Regan, T. S., Kleinke, J., & Arnold, S. (2009). Private health plans perspectives: Electronic personal health records and electronic prescribing. American Health & Drug Benefits, 2(6), 252-253.

McGrath, J. M., Arar, N. H., & Pugh, J. A. (2007). The influence of electronic medical record usage on nonverbal communication in the medical interview. Health Informatics Journal, 13(2), 105-118. Available at: https://doi.org/10.1177/1460458207076466.

Moseley, S., Randall, S., & Wiles, A. (2003). Experience within ETSI of the combined roles of conformance testing and interoperability testing. Paper presented at the Standardization and Innovation in Information Technology, 2003. The 3rd Conference on.

Mullarkey, M. T., & Hevner, A. R. (2019). An elaborated action design research process model. European Journal of Information Systems, 28(1), 6-20. Available at: https://doi.org/10.1080/0960085x.2018.1451811.

National Department of Health. (2012). eHealth strategy South Africa 2012-2016. Retrieved from: http://www.health.gov.za/index.php/2014-08-15-12-54-26/category/95-2012s?download=171:ehealth-strategy-south-africa-2012-2016 .

National Department of Health & Council for Scientific and Industrial Research. (2014). Health nmative sandards famework for ehalth iteroperability. Retrieved from: http://www.samed.org.za/Filemanager/userfiles/hnsf-complete-version.pdf .

National Planning Commission. (2012). National development plan 2030: Our future–make it work. Presidency of South Africa, Pretoria, 1. Retrieved from: https://www.gov.za/documents/national-development-plan-2030-our-future-make-it-work .

NDoH, & CSIR. (2014). South African national health normative standards framework for interoperability in ehealth. Government Gazette Version. Retrieved from: http://www.samed.org.za/Filemanager/userfiles/hnsf-complete-version.pdf .

NHTA. (2007). Interoperability framework V2.0. National eHealth Transition Authority. Retrieved from: https://www.ghdonline.org/uploads/NEHTA_2007_-_InteroperabilityFramework.pdf .

Nielsen, J. (2000). Why you only need to test with 5 users. Useit. com Alertbox.

Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(1), 53.

Oemig, F., & Snelick, R. (2016). Healthcare interoperability standards compliance handbook. Cham, the Netherlands: Springer International Publishing.

Oh, H., Rizo, C., Enkin, M., & Jadad, A. (2005). What is ehealth (3): A systematic review of published definitions. Journal of Medical Internet Research, 7(1), e1.

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45 - 77.

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2008). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45-77.

Pinelle, D., & Gutwin, C. (2002). Groupware walkthrough: Adding context to groupware usability evaluation. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., & Lindgren, R. (2011). Action design research MIS quarterly. 35(1), 37 - 56.

Simon, H. A. (1996). The sciences of the artificial (3rt ed.): Cambridge, Massachusetts: MIT Press.

Stewart, B. A., Fernandes, S., Rodriguez-Huertas, E., & Landzberg, M. (2010). A preliminary look at duplicate testing associated with lack of electronic health record interoperability for transferred patients. Journal of the American Medical Informatics Association, 17(3), 341-344. Available at: https://doi.org/10.1136/jamia.2009.001750.

Terry, N. (2014). Pit crews with computers: Can health information technology fix fragmented care? Retrieved from: https://www.ssrn.com/abstract=2383470 .

Thomas, R., & Velaphi, S. (2014). Abuse of antiretroviral drugs combined with addictive drugs by pregnant women is associated with adverse effects in infants and risk of resistance. South African Journal of Child Health, 8(2), 78-79. Available at: https://doi.org/10.7196/sajch.734.

Vernadat, F. B. (2007). Interoperable enterprise systems: Principles, concepts, and methods. Annual Reviews in Control, 31(1), 137-145. Available at: https://doi.org/10.1016/j.arcontrol.2007.03.004.

Wager, K. A., Lee, F. W., & Glaser, J. P. (2017). Health care information systems: A practical approach for health care management: John Wiley & Sons.

WHO. (2005). Fifty-eigth world health assembly. Geneva. Retrieved from: https://apps.who.int/gb/ebwha/pdf_files/WHA58-REC1/english/A58_2005_REC1-en.pdf.

WHO. (2018). Classification of digital health interventions v1.0 A shared language to describe the uses of digital technology for health. Retrieved from: http://apps.who.int/iris/bitstream/10665/260480/1/WHO-RHR-18.06-eng.pdf?ua=1 .

WHO. (2019). ehealth at WHO. Retrieved from: https://www.who.int/ehealth/about/en/ .

World Health Organization. (2012). National ehealth strategy toolkit. International Telecommunication Union. Retrieved from: https://www.who.int/ehealth/publications/overview.pdf .

Views and opinions expressed in this article are the views and opinions of the author(s), International Journal of Business, Economics and Management shall not be responsible or answerable for any loss, damage or liability etc. caused in relation to/arising out of the use of the content.

Footnote:

1. Considering the nature of this research, a detailed explanation of the research methodology is necessary.