Introduction

The COVID-19 pandemic forced restrictions on freedoms, significantly impacting health services and preventing adequate and timely clinical care delivery. This context prompted the expansion of telemedicine tools implemented to bridge this care gap. As part of this effort, healthcare professionals faced new challenges arising from using technologies with varying levels of knowledge and familiarity.

According to the Association of American Medical Colleges (AAMC), the use of telemedicine was already growing steadily in healthcare institutions even before the COVID-19 pandemic [1]. However, in Europe, such interest in technology implementation has not been matched by adequate education and training for staff. Most healthcare professionals reported insufficient training in telemedicine tools as of 2016 [2].

Following the above, in Chile, measures focused on promoting telemedicine have been progressively incorporated since 2018 by implementing the National Telehealth Program. These are part of the National Health Strategy and aim to mitigate geographical gaps and improve healthcare quality, process efficiency, and waiting times [3].

A recent collaborative study concludes that Chile has not reached the expected development in telemedicine by 2022. Consequently, it identifies five priority areas that need strengthening: regulatory framework, good clinical practices, human resources, infrastructure, and financing [4]. Concerning human resources, although it is important for professionals to acquire new knowledge, competencies, and skills in digital health, to date, there is no information available to determine the number of people trained in this field, their place of performance, or the degree of preparation of technicians and professionals in digital health [4].

Villarroel [5], in a qualitative study conducted among health professionals in southern Chile, concluded that the COVID-19 pandemic significantly boosted the use of telemedicine, demonstrating its potential to improve access to healthcare, especially in regions with limited infrastructure. However, the effectiveness and benefit of this telemedicine strategy will depend on the value that professionals place on it [5]. Avendaño-Veloso et al. [6] highlight the consolidation of telemedicine as a key tool for rapid response to emergencies, such as the COVID-19 pandemic. However, challenges related to digital equity and the training of health professionals in this remote care strategy persist. There is agreement that the pandemic marked a turning point for telehealth and telemedicine in Chile, highlighting the need to strengthen its implementation in a sustainable and equitable manner and train professionals in this area.

Within this framework, the Association of American Medical Colleges has defined a series of telemedicine competency domains for the education and training of future healthcare professionals. The telemedicine competency model this association defines involves six domains, as shown in Table 1.

The drastic change caused by the COVID-19 pandemic led many health institutions to integrate telemedicine and caused inexperienced professionals to become involved in health care through digital means. One of these telemedicine experiences was carried out at the Ann & Robert H. Lurie Pediatric Hospital in Chicago [8]. In this experience, 305 health professionals responded to a survey about their perception and attitude towards telemedicine, having been previously trained to provide health care through technology. The main concerns included the reliability of internet services (65%), limitations in performing a physical examination of the patient (52.5%), the difficulty for families to access video services due to lack of digital devices, cellular data, or wifi (51.4%), and poor audio or video quality (74.6%). These percentages represent the highest concerns recorded in the study, augmented by patient experience.

The aforementioned study [8] raises two central questions about the telemedicine implementation experience. First, how prepared are patients in a community to receive care through digital means? Second, how prepared are the healthcare institution and its staff to provide such care? Despite the development of telemedicine competency-based training models, few tools are currently available to assess these questions. Conceptual models of telemedicine competencies have been developed to emphasize the telemedicine education process rather than as assessment guidelines through on-site observation or self-report questionnaires.

Two key instruments in the field of telemedicine can be considered benchmarks. One is the Teaching Interpersonal Skills for Telehealth Checklist (TIPS-TC) [9], and the second is the Telehealth Readiness Assessment Tool [10]. The TIPS-TC checklist allows educators or administrators to assess observable interpersonal skills behaviors in health care via telemedicine. This instrument assesses health care readiness, verbal and nonverbal communication, relationship building, and environmental management. It applies to evaluating various disciplines and roles within health care and helps evaluate skills in general health care.

On the other hand, the Telehealth Readiness Assessment Tool is designed to help physician practices assess their readiness for telemedicine implementation. This instrument covers five domains: essential readiness, financial considerations, operations, team engagement, and patient readiness. In interpreting its results, a score of less than 50% indicates low readiness, between 50% and 75% indicates moderate readiness and greater than 75% indicates high readiness for telemedicine implementation.

Integrating the information presented, it can be concluded that neither of the two measurement instruments mentioned above possesses the conceptual breadth of the Association of American Medical Colleges' telemedicine competency model [1,4]. In addition, both instruments are completed by third parties, not by the individuals being assessed. In the case of the TIPS-TC, it is an educator or administrator who assesses the interpersonal skills of a healthcare professional during care through digital means. The Telehealth Readiness Assessment Tool is designed to be completed by the managers of a healthcare institution. This is because it evaluates the operational and financial aspects of telemedicine implementation. Thus, taking the Association of American Medical Colleges competency model as a theoretical reference, neither of the two instruments is designed to assess these competencies comprehensively.

For some years, implementing competency-based medical education has been advocated, which has presented challenges for its correct evaluation. According to Miller’s pyramid, the evaluation stages of competency are from lowest to highest: knowing, knowing how, showing how, and doing [11]. Of these four stages, self-report questionnaires can only be used in the first stage, which is related to the level of knowing. That is, the most basic level of competency, which refers only to knowledge, can be assessed through self-report scales.

Within this framework of self-report questionnaires, scales proposed for the self-report assessment of constructs associated with telemedicine, designed in other countries, were reviewed. Nayar, McFarland, Chandak, and Gupta [12] applied a modification of the Practitioner Telehealth Readiness Assessment Tool into a 16-item self-report instrument applicable to oral health professionals. The study suggests a factorial structure of three dimensions consistent with the original model. Another instrument used in teleodontology is a 26-item Likert scale that assesses four dimensions: the usefulness of teleodontology for dental practice, its usefulness to patients, the potential of teleodontology to improve clinical practice, and concerns related to its use [13]. In another study conducted in Saudi Arabia [14], this same 26-item scale was adapted, but different dimensions were stated: knowledge of teleodontology, practice, attitudes, and training. This study found that oral health professionals have poor knowledge, practice, and training in teleodontology. However, a positive disposition towards its implementation stands out at the attitudinal level.

The Knowledge, Skills, and Attitudes (KSAs) scale was applied to nursing professionals in the Netherlands, which measures through 31 items the level of confidence with which they perceive their knowledge, skills, and attitudes to apply telemedicine. This study identified a medium level of knowledge, skills, and attitudes toward telemedicine and recommends strengthening training in this area so that professionals feel more confident and prepared to practice telemedicine [15]. Another instrument used to identify knowledge, confidence, and attitudes towards telemedicine was the Telemedicine Objective Structured Clinical Exam (TeleOSCE) scale, which is composed of 22 items and is applied to nursing professionals in Indonesia. This instrument identified that 76% of the nursing professionals surveyed have a moderate level of knowledge, and 61.5% perceive themselves prepared to practice telemedicine [16].

Considering these self-report scales, it can be concluded that, to date, there is no general instrument for assessing telemedicine competencies nor a consensus on how to assess them through self-reports. The dimensions that make up the construct are not clear. At the conceptual and operational definition level, various constructs are handled, such as telemedicine readiness, practical knowledge, and attitudes about telemedicine and teleodontology, and interpersonal skills for telemedicine.

Some instruments reviewed have not been validated by exploratory or confirmatory factor analysis. Nor do they adequately report the reliability levels of their scales. Nevertheless, they constitute important inputs for proposing a new instrument adapted to the Chilean reality. In this line, the objective of the present study was to develop and evaluate the reliability and validity of an instrument designed to measure the perception of health professionals and technicians regarding their competencies and level of preparation in telehealth.

Methods

An instrumental study [17] was conducted to develop and validate an instrument of our own creation that measures the perception of competencies and level of preparedness for telehealth. For the development of the study, the three steps recommended by Lambert and Newman [18] were followed, which include:

1. Contextualization and definition of the construct through literature review.

2. Operationalization of the construct includes the development of the items and content validation.

3. Evidence of construct validity through empirical testing.

The application of the instrument in its content validation and data collection stages for construct validity and reliability was carried out through an e-mail invitation to participate in the study. The self-report link to answer the scale using the SurveyMonkey platform in 2023 was attached to the e-mail. Before applying the scale, participants were asked to sign the informed consent form and were informed of the anonymity, confidentiality, benefits, and risks of participation. This study was certified by the Ethics Committee of the Universidad Católica de la Santísima Concepción (N° 60/2022).

Instrument

The instrument used in the surveys was developed within the framework of the Innovation Fund for Regional Competitiveness (FIC-R-2021) of the Biobío Regional Government. The instrument was designed for health professionals and technicians, using the inclusion criterion that they have at least one year’s experience in telematic health care as an inclusion criterion. The exclusion criterion was that they were health professionals and/ or technicians without telematic health care experience.

The preliminary version of the instrument was based on the proposal for telemedicine competencies of the Association of American Medical Colleges [1]. Thus, 51 items were obtained, grouped into eight dimensions: telehealth applicability (4 items), clinical approach in telehealth (7 items), communication in telehealth (11 items), networking and management in telehealth (7 items), excellence in telehealth (4 items), information technologies in telehealth (3 items), patient safety and privacy (4 items), and perception of the level of preparation in telehealth (11 items). In the identified dimensions, items were written based on the observable behaviors, knowledge, and skills described in the Association of American Medical Colleges competency proposal [1]. This study used the reflexive method to ensure the relevance and validity of the scale items, as it was based on a critical and contextualized analysis of the phenomenon to be evaluated [19].

This preliminary version was subjected to content validity by experts and subsequently to construct validation. To this end, it was answered by the participants, who had to score each item on a five-point Likert scale according to their degree of agreement with the statements presented. The response options were: strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree.

Content validity

The expert judgment method was used to obtain content validity, which is frequently employed. This gives the degree of agreement between different experts concerning the items of an instrument based on their relevance, representativeness, and coherence between the items and the contents to be evaluated [20,21].

Non-probability purposive sampling was used to select the experts [22]. In addition, elements such as the experts' academic training, being a health service professional, having at least one year of experience in telehealth, and being recognized in the community were considered [23]. As an exclusion criterion, health professionals and/or technicians with no experience in telemedicine care were considered. After identifying these experts, an invitation to participate in the process was sent via e-mail, detailing the study’s objectives and the protocol for responding, together with the proposed 51-item scale. In this protocol, the experts had to evaluate each item, indicating whether it met the criteria measured (relevance, representativeness, and coherence) on a scale of 1 to 3, considering the item’s relationship with the dimension to which it belonged, with 3 being the maximum evaluation.

The participating information-producing sample consisted of seven experts, thus meeting the criteria of McGartland et al. [24] of between six and 20 judges. To analyze the responses provided by the experts, the item ratings were entered into an Excel spreadsheet. Then, the content validity analysis was performed using the calculation of the content validity ratio (CVR) and the content validity index (CVI) proposed by Lawshe [25], currently considered the most appropriate given their numerous advantages, such as the contribution of the item and instrument level, and the focus on the experts' agreement regarding the relevance of the item [26].

Construct validity and reliability

The instrument obtained from the content validation was applied to health professionals and technicians from the Biobío Region for the construct validity and reliability study. The inclusion criterion was that they had experience (at least one year) in telematic health care. The exclusion criterion was to be a health professional and/or technician without experience in telemedicine and telehealth. A non-probabilistic purposive sampling was used [25,27], determined according to the availability of participants who met the established criteria. An effective total of 83 respondents [28] was generated, of which 55.4% were women, 43.4% were men, and 1.2% identified with another gender. The nationality of the participants was mainly Chilean (96.4%), followed by 2.4% Venezuelan and 1.2% Colombian. These proportions by sex and nationality are close to the data reported by the Superintendence of Health in the working document "Characterization of health professionals in Chile 2021" [29]. This publication identifies that 69.8% of professionals are women and 30.2% are men. Likewise, the percentage of registered foreign health professionals is 5.8% of the total number of professionals in this area in Chile, with Venezuelans being the most representative [29]. There are varied professions and occupations (which can be seen in Figure 1), and the performance of functions is mainly in the urban sector (91.6%), in contrast to the rural sector (8.4%). Finally, only 20.5% of the respondents report having some type of training or education in telehealth, while 79.5% have no such experience.

Figure 1

Percentages associated with the occupations of those who responded to the instrument.

Data was collected from health professionals and technicians who responded to the instrument in the Biobío Region, Chile, 2023.
Source: Prepared by the authors based on the results of the study.

Full size

Construct validity was performed through an exploratory factor analysis since this method is recommended for assessing construct validity when the scale is new or its structure is unknown [30]. Bartlett’s test of sphericity and the Kaiser-Meyer-Olguín statistic (KMO) was performed to assess the relevance of the exploratory factor analysis on the instrument. Likewise, the generation of the anti-image correlation matrix was added, following current recommendations [30].

Factor extraction was performed using the principal axis method, which performs best when the sample is relatively small, i.e., less than 300 subjects [31], and when Likert-type scales are used [32]. Horn’s parallel analysis was used to estimate the number of factors, empirically considered one of the most accurate methods [32], in combination with the screen test, which is perceived as very useful to complement the interpretation of the empirical estimators [33,34].

Finally, the rotation of the factorial solution was performed using the Oblimin oblique method, as this type of method is currently highly recommended compared to orthogonal rotations [31,35]. The rotated matrix was interpreted using the Hair criterion [28] for samples of 85 subjects so that all items with factor loadings above 0.6 were retained. The scale’s reliability was obtained by calculating Cronbach’s α coefficient, which measures the consistency of the scale, i.e., whether the items produce stable and reproducible results.

Results

From validating the expert judgment and obtaining the content validity ratio values for each item, it was possible to identify the items with the lowest scores, obtaining 19 items below 0.7. We eliminated 16 of them, keeping the three items of the "Privacy and patient safety" dimension since this would be eliminated following the quantitative criterion alone. Given the importance of this dimension in the objective of the scale as a whole, it was decided to keep these items. This resulted in a 35-item scale with a content validity index of 0.77. This value meets the threshold established by Tilden, Nelson & and May [36] of 0.7, while it is close to the suggestion of 0.8 established by Davis [37].

To determine whether factors were present in the scale, three statistics were used in a complementary manner:

1. Bartlett’s test of sphericity generated a statistically significant result, X2 (465,000) = 3664.13; p < 0.001.

2. The Kaiser-Meyer-Olguín statistic presented a value of 0.911, which is considered by Kaiser [38] as an ideal value.

3. The anti-image matrix indicated that the diagonal values exceed the suggested threshold (0.5) [30].

The factor analysis indicated two factors based on Horn’s parallel analysis and the sedimentation plot. Specifically, the parallel analysis indicated the presence of two factors with higher eigenvalues (17.772; 5.183) than those obtained randomly (1.516; 1.178). This agrees with the sedimentation plot (Figure 2), which details the presence of two factors. This two-factor model explained 75.5% of the total variance, which exceeds the recommended threshold of 60% [28].

Figure 2

Sedimentation plot with observed and simulated eigenvalues calculated from Horn’s parallel analysis.

Source: Prepared by the authors of this study.

Full size

Subsequently, the factorial solution was rotated using the Oblimin oblique method. During the first rotation, the items that presented a factorial loading below the threshold of 0.6 were eliminated. Then, the factorial solution was rotated again, obtaining all the remaining items with loadings above the target threshold.

As can be seen in Table 2, Factor 1 (perception of telehealth competencies) is composed of 19 items ordered from highest to lowest loadings (items 3, 10, 4, 9, 11, 8, 5, 7, 14, 15, 12, 13, 6, 1, 18, 2, 19, 20 and 17). The analysis of its internal consistency indicates a Cronbach’s α coefficient of 0.98, with correlations between the items and the corrected total from r = 0.93 (item 10) to r = 0.73 (item 17). These items correspond to those assigned initially to the dimensions applicability of telehealth consultation, clinical approach in telehealth, telehealth communication, telehealth networking and management, excellence, technologies and privacy, and patient safety.

On the other hand, Factor 2 (perception of the level of preparedness in telehealth) is composed of 11 items ordered from highest to lowest loadings (items 24, 30, 31, 27, 29, 23, 25, 21, 28, 22, and 26). The analysis of its internal consistency indicates a Cronbach’s α coefficient of 0.97, with correlations between the items and the corrected total ranging from r = 0.93 (item 24) to r = 0.81 (item 28). All the items that make up this factor correspond to those originally assigned to the dimension of perceived level of preparedness in telehealth.

The fact that both factors obtained a high Cronbach’s α coefficient (0.98 and 0.97) indicates that the items are strongly related to each other and measure the dimension or factor in which they are positioned in the exploratory factor analysis result.

Each item was scored on a Likert-type scale:

1. Strongly disagree

2. Disagree

3. Neither agree nor disagree

4. Agree

5. Strongly agree

The scale has 150 points (Factor 1 = 95 points; Factor 2 = 55 points). For the interpretation of the scores, a division of categories by terciles was used (Table 3).

After consolidating the factors and their items, descriptive statistics were obtained for both factors (Table 4). Finally, the relationship between both factors was calculated. This analysis indicates that the relationship between factors is direct and statistically significant (p < 0.001), with a Pearson correlation coefficient of 0.48, which is considered a moderate correlation [39].

Discussion

The telematic delivery of health care is a tool that benefits the recipients of the Chilean health system, increasing the coverage of various health services. In this context, it is essential to have instruments to identify, from the professionals and technicians themselves, what competencies and levels of preparation are perceived in the deployment of quality telehealth care.

The present study focused on developing and validating an instrument that shows how health professionals and technicians in the Biobío Region perceive their own competencies and level of preparation with respect to the delivery of care telematically. In the sample analyzed, the results of the instrument showed satisfactory psychometric properties, with evidence of validity and reliability to assess the constructs of perception of competencies and level of preparedness in telehealth.

Although the instrument was designed based on eight dimensions, it empirically demonstrates that it comprises two factors. This result is supported by both Horn’s parallel analysis and the sedimentation plot. The decision to keep two factors as the factorial solution is due to two key aspects:

1. The principle of parsimony that should guide the factor estimation stage states that the simplest solution should always be determined. That is, with the smallest possible number of factors [40].

2. The behavior of the extracted factors.

Concerning this last point, it is possible to appreciate that both factors can be considered reliable to the extent that they have four or more loadings above 0.6 [41]. All the items load on this threshold is a good indicator of sample fit as it meet the criterion established by Hair [28] for samples of 85 subjects. Similarly, according to Costello & Osborne [32], small samples can produce accurate analyses when uniformly high communalities (between 0.4 and 0.7). After extraction, it can be observed that all items have commonalities within this range. Also, if all factor loadings exceed 0.6, factor analyses can be conducted with small sample sizes [42,43]. On the other hand, both extracted factors meet the criteria established by Watkins [31], which would allow estimating a factorial solution as adequate:

1. At least three variables in each factor load on the threshold (in this case, on 0.6).

2. There are no cross-loadings.

3. Each factor has internal consistency reliability above 0.7.

4. The factors make theoretical sense.

However, as mentioned at the beginning of this section, the original scale was composed of eight dimensions, and the scale empirically demonstrates that only two explain the variance of the items. Thus, to make theoretical sense, Factor 1 was called "perception of telehealth competencies", understanding that, although there are seven domains, they operate in a cross-cutting manner in practice. Thus, for example, the elements that point to telehealth communication should be present throughout telehealth care, permeating other dimensions such as patient safety, networking and information technologies. Thus, this factor points to the actions professionals can currently deploy in the telehealth framework. On the other hand, Factor 2 brings together all the items designed initially under the dimension "perception of the level of preparedness in telehealth". This comprises all the items that evaluate how prepared professionals and technicians feel to develop the activities required in this care modality.

Regarding the behavior of the factors, it is relevant to highlight that both correlate appropriately, with a moderate Pearson correlation (r = 0.48). According to several authors, correlations should be between 0.3 and 0.7 to indicate that both factors measure the same phenomenon without redundancy [44,45].

In summary, the instrument meets the necessary conditions to affirm its validity and reliability in measuring two relevant constructs: the perception of competencies and the perception of the level of preparedness to provide telehealth care. This is based on the results obtained from the sample. This is highly relevant in light of the descriptive results obtained in this research, which indicate a large majority of respondents who report having no education or training in this area. This is consistent with the realities reported by studies in other countries [2]. In this context, the results of the application of this instrument will make it possible to identify strengths and weaknesses in the training of those who perform telemedicine health care, and to focus on the needs for updating in the various populations.

Using scales such as the one developed and validated in a Chilean population in this study will make it possible to measure competencies and perceptions of telemedicine and telehealth preparedness in health services. Its importance lies in identifying gaps in knowledge, skills, and confidence of health professionals who need to use this modality. With this, it is possible to guide decisions regarding the lines of training implemented in the services and institutions that train health professionals. Initiatives such as the digital health strategy in Chile show the importance of strengthening the digital capabilities of healthcare personnel to ensure the effectiveness of telemedicine and other digital services [46]. Globally, this approach is aligned with the World Health Organization (WHO) strategies that highlight digital equity and training as key pillars to close the technological gap between regions [47]. However, competency analysis also poses challenges, such as developing inclusive policies that address technological inequalities between countries. Properly implementing this type of scale can improve global capabilities and strengthen international cooperation in e-health.

The study has limitations related to the sample size, both in number and geographic representativeness, as the data were obtained from a sample restricted to a specific region. This limits the generalization of the findings to other areas of the country with different telehealth contexts. In addition, there is a need to broaden the diversity of healthcare disciplines included in the study, as the current sample may not fully reflect the perceptions and needs of all specialties.

Recommendations

Two main lines of future analysis are suggested. First, although the sample size did not impede obtaining reliable and trustworthy results, it is recommended that the analysis be continued to strengthen the instrument’s construct validity. The process should be done by increasing the sample to include professionals from other regions of the country, broadening the diversity of health disciplines, improving the interpretation of the scale scores, and incorporating confirmatory factor analysis to consolidate the findings obtained in the exploratory analysis.

On the other hand, it is suggested that the information that can be obtained through this instrument be complemented by perceptions of other key actors in the health system, such as patients and authorities, as well as descriptive comparative studies between sociodemographic variables. By triangulating this information, it will be possible to obtain important inputs that will make it possible to focus the training of health professionals and technicians and to adapt the scale to other realities. This, always in pursuit of improving the quality of care, ensuring patient privacy, standardizing some practices, and increasing the coverage of our country’s health system.

Conclusion

Despite the small sample size, the instrument has adequate psychometric indicators to measure the perception of competencies and level of preparation in telehealth in health professionals and technicians of the Biobío Region.

In addition, the two factors that make up the instrument allow the identification of training and education needs in telehealth for professionals in the area.

Through the application of this tool, it is possible to guide the decisions and guidelines of the training implemented in the services and institutions that train health professionals.