Development and Validation of a Tool to Evaluate Dental Residents’ Understanding and Satisfaction with Their Postgraduate Exit Summative Assessment

By Syed Murtaza Raza Kazmi¹, Fakeha Azhar Syed¹, Tabassum Zehra²

Affiliations

1. Department of Dentistry and Oral Health Sciences, The Aga Khan University Hospital, Karachi, Pakistan
2. Department of Educational Development, The Aga Khan University, Karachi, Pakistan

doi: 10.29271/jcpsp.2025.12.1629

ABSTRACT
Students’ comprehension of the assessment process is critical for effective evaluation. The authors at the Aga Khan University Hospital developed a questionnaire to assess Pakistan’s dental postgraduate residents' understanding and satisfaction with the summative assessment system conducted by the College of Physicians and Surgeons Pakistan. Following ethical approval, six senior dental faculty and one medical educationist reviewed and validated the 37-item tool across seven domains from September to November 2023. The scale-level content validity index for relevance and clarity was 0.97, with a modified Kappa statistic of ≥0.85. All items received item-level content validity index  ≥0.86 except one regarding “examiner calibration” (0.71) which was retained based on literature support. Internal reliability testing with 15 participants yielded a Cronbach’s alpha of 0.7. This reliable and validated tool can be used to evaluate learners’ perceptions and satisfaction with assessment procedures and is applicable across postgraduate dental education specialities.

Key Words: Competency-based learning, Dental education, Postgraduate education, Tool validation, Assessment system, Dental residents, Content validity, Internal reliability, Summative assessment, Curriculum alignment.

Contemporary pedagogy supports competency-based learning (CBL), a student-centred approach focusing on learning out- comes.¹ Postgraduate (PG) training in health sciences aims to develop competent clinicians and academicians for society and encompasses cognitive and psychomotor domains with clinical reasoning.² Assessment, an integral part of PG training, should effectively evaluate professionals' clinical knowledge and skills acquired during training, thereby promoting a coherent educational system.^1,3 Teachers can identify areas for improvement in trainees and provide a clear understanding of expectations and grade performance through evaluation.³ Understanding the relevance and purpose of assessments in relation to learning goals boosts student engagement and motivation.

Assessments are linked to competency-based curriculum objectives, improving the educational experience by providing constructive feedback and promoting students’ development and achievements.^1,3

 Through effective teacher-learner relationships, learners gain insights into evaluation standards, adjust their learning tactics, and prioritise resources, thereby enhancing their under- standing of the assessment process.

The authors conducted a session to familiarise prosthodontics trainees and exit (summative) examination candidates with the assessment system undertaken by the College of Physicians and Surgeons of Pakistan (CPSP). Using a short semi-structured questionnaire form, post-session feedback and suggestions were anonymously gathered from participants regarding their satisfaction with the assessment process and their perceptions and understanding of the significance of each component of the evaluation system. Alarmingly, the results showed their limited knowledge and trust in the current examination process (Table I). These findings necessitated further exploration, leading the authors to develop and validate a tool to explore the perceptions of postgraduate (dental) trainees and exit-examination candidates regarding the importance and transparency of each assessment component. Slavec and Drnovsek proposed ten steps for tool development and its validation. However, the authors followed the initial six steps in this study.⁴

Step 1 involved content domain specification. Five key domains were identified to evaluate trainees' perceptions of the FCPS exit (summative) examination, based on responses and impressions gathered during the debriefing session with trainees.

Table I: Results of the debriefing session with FCPS-II clinical examination candidates evaluating their satisfaction, understanding, perceptions, and suggestions.

Evaluations	Candidates appeared in clinical examination*
	Yes  11 (55%)	No  9 (45%)	p-values
Are you satisfied with the quality of questions in theory examinations?	-	-	-
Yes	8 (72.7%)	6 (66.7%)	>0.99
No	3 (27.3%)	3 (33.3%)
Are you satisfied with the current pattern of clinical examination?	-	-	-
Yes	5 (45.5%)	5 (55.6%)	>0.99
No	6 (54.5%)	4 (44.4%)
Do you think there must be hands-on tasks in the clinical examination?	-	-	-
Yes	1 (9.1%)	2 (22.2%)	0.280
No	9 (81.8%)	4 (44.4%)
Maybe	0	3 (33.3%)
Candidates’ understanding of the examination components	-	-	-
1. Short answer questions (SAQs)	-	-	-
Assess the depth of the knowledge	9 (81.8%)	5 (55.6%)	0.086
Assess the width of the knowledge	0	2 (22.2%)
Easy for examiners to develop examination paper	2 (18.2%)	0
Legacy of the examination system	0	2 (22.2%)
2. Multiple choice questions (MCQs)	-	-	-
Assess the depth of the knowledge	5 (45.5%)	4 (44.4%)	>0.99
Assess the width of the knowledge	3 (27.3%)	2 (22.2%)
Easy for examiners to develop examination paper	1 (9.1%)	1 (11.1%)
Promote modernisation of the examination system	2 (18.2%)	2 (22.2%)
3. Viva	-	-	-
Assess the depth of the knowledge	7 (63.6%)	9 (100%)	0.230
Assess the width of the knowledge	2 (18.2%)	0
Easy for examiners to develop examination paper	2 (18.2%)	0
Promote modernisation of the examination system	0	0
4. Short case	-	-	-
Assess the depth of the knowledge	1 (9.1%)	1 (11.1%)	>0.99
Assess the width of the knowledge	1 (9.1%)	1 (11.1%)
Correlate theoretical knowledge with clinical scenarios	9 (81.8%)	7 (77.8%)
Easy for examiners to develop examination paper	0	0
Promotes modernisation of the examination system	0	0
Candidates’ perceptions regarding examination components
1. Multiple choice questions (MCQs)
Not applicable	0	0	0.370
Redundant	0	0
Need alternative	0	0
Important	7 (63.6%)	3 (33.3%)
Very important	4 (36.4%)	6 (66.7%)
2. Short answer questions (SAQs)
Not applicable	2 (18.2%)	0	0.611
Redundant	0	1 (11.1%)
Need alternative	3 (27.3%)	4 (44.4%)
Important	4 (36.4%)	2 (22.2%)
Very important	2 (18.2%)	2 (22.2%)
3. Viva
Not applicable	0	0	0.267
Redundant	0	0
Need alternative	1 (9.1%)	3 (33.3%)
Important	6 (54.5%)	2 (22.2%)
Very important	4 (36.4%)	4 (44.4%)
4. Short case
Not applicable	0	0	0.657
Redundant	0	0
Need alternative	2 (18.2%)	0
Important	5 (45.5%)	5 (55.6%)
Very important	4 (36.4%)	4 (44.4%)
5. Long case
Not applicable	0	0	0.850
Redundant	0	0
Need alternative	4 (36.4%)	2 (22.2%)
Important	5 (45.5%)	4 (44.4%)
Very important	2 (18.2%)	3 (33.3%)
Viva and short cases should be replaced by TOACS
Strongly disagree	1 (9.1%)	0	0.648
Disagree	2 (18.2%)	0
Neutral	3 (27.3%)	5 (55.6%)
Agree	2 (18.2%)	2 (22.2%)
Strongly agree	3 (27.3%)	2 (22.2%)
Viva should be replaced b TOACS
Strongly disagree	1 (9.1%)	0	0.374           0.925
Disagree	3 (27.3%)	0
Neutral	3 (27.3%)	3 (33.3%)
Agree	1 (9.1%)	3 (33.3%)
Strongly agree	3 (27.3%)	3 (33.3%)
Long case should be replaced by TOACS	-	-
Strongly disagree	0	0
Disagree	2 (18.2%)	1 (11.1%)
Neutral	3 (27.3%)	4 (44.4%)
Agree	2 (18.2%)	2 (22.2%)
Strongly agree	4 (36.4%)	2 (22.2%)
Data are given in numbers (n) and percentages (%). p-value, obtained using Fisher's exact test and analysed using SPSS version 23.0, were used to determine the difference in candidates' responses regarding their satisfaction with the examination prorcess, their understanding of the purpose of each examination component, and their perceptions of the significance of each component. A five-point Likert-type scale was used to record participants’ level of agreement with the suggestion to replace TOACS with certain clinical examination components, coded as follows: 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree. *Clearing the theory examination was a prerequisite for taking the clinical examination. A p-value <0.05 was considered statistically significant. FCPS-II refers to the summative exit examination for the fellows of the College of Physicians and Surgeons Pakistan. TOACS: Task-oriented assessment of clinical skills.

Table II: Results of content validation of the developed questionnaire.
 

Relevance				Clarity				Interpretations
Agreement	I-CVI	Pc	k	Agreement	I-CVI	Pc	k
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
5/7	0.71	0.164	0.66	5/7	0.71	0.164	0.66	Good
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	6/6	1.00	0.016	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1	7/7	1.00	0.008	1	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
6/7	0.86	0.055	0.85	6/7	0.86	0.055	0.85	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
7/7	1.00	0.008	1.00	6/6	1.00	0.016	1	Excellent
7/7	1.00	0.008	1.00	7/7	1.00	0.008	1	Excellent
S-CVI = 0.97 (95% CI)				0.97 (95% CI)
I-CVI: Pc = Chance agreement, k = Modified Kappa

These included satisfaction with the current examination pattern or suggestions for improvements, agreement with recently proposed changes, inclusion of hands-on tasks, significance and effectiveness of each assessment component, and post-examination feedback. Collectively, these domains provide a comprehensive framework for understanding of the trainees’ views and enhancing the summative examination process.

Step 2 involved item pool generation. Items in the developed questionnaire were designed to ensure comprehensive coverage of the five outlined domains. Both closed-ended and open-ended items were used in this tool to record participants’ perceptions and suggestions. Eight questions (22-29) were adapted from a published study conducted by Graham et al.⁵ Caution was taken to balance the number of items across the domains to maintain questionnaire length and participant engagement.

Step 3 involved content validity generation. The 37-item questionnaire was shared via REDCap software version 13.9.3 with two medical educationalists and ten dental faculty (two each from prosthodontics, periodontics, orthodontics, operative dentistry, endodontics, and oral maxillofacial surgery) to validate the tool for relevance and clarity of each item. One medical educationalist and six dental faculty members partici-pated in the tool validation process.

Step 4 involved questionnaire development and evaluation. Data analysis for content validity of the questionnaire was conducted using Google Sheets. Item and scale-level content validity on average (I-CVI and S-CVI/Ave) were calculated for both relevance and clarity. The proportion of chance agreement (Pc) was determined, and modified Kappa (k) statistics were computed for each item in the questionnaire to verify the ICV-I score.⁶

All items achieved acceptable I-CVI scores of 0.86 or higher, with k values of ≥0.85, interpreted as excellent. The S-CVI/ Ave was calculated as 0.97 for both relevance and clarity. One open-ended question addressing the effect of examiner cali-bration on the candidate’s assessment process, however, received an I-CVI of 0.71 (k = 0.66; interpreted as good) for both relevance and clarity (Table II). The item was modified to improve its clarity and retained in the final questionnaire based on strong support from the literature.

Step 5 involved translation and back translation. Although the added eight questions were in the English language, they were restructured according to the context of the currently developed tool, thus ensuring better clarity and relevance.

Step 6 involved a pilot study. The authors invited twenty-two residents and examination candidates from three dental specialities to participate in the pretesting of a questionnaire. Fifteen participants consented to the pilot study, and the questionnaire's internal reliability was calculated using IBM SPSS Statistics Software version 23.0, yielding a Cronbach's alpha of 0.7. While this value was acceptable, a higher value would have been preferable.

This study developed and validated the 37-item question-naire, representing a significant step towards assessing residents' perceptions of the examination system in dental postgraduation assessment. This tool has the potential to provide reliable insights into trainees’ perceptions regarding assessment practices not only within the CPSP examinations in dentistry but also across various healthcare educational settings after required modification. Content validity and reliability are only one aspect of ensuring the overall quality of the measurement tool. Furthermore, the small sample size used in the pilot study of the questionnaire may have hampered the breadth and generalisability of the findings. Future studies should examine additional psychometric qualities, such as construct validity, to increase the tool’s utility and resilience. These findings lay the groundwork for future research to explore trainees' understanding and perceptions of existing examination systems, thereby facilitating continuous improvement in dental education practices.

The authors also suggested that future studies should include a large and diverse participant pool to better examine the population’s general attitudes regarding the concept thoroughly.

ETHICAL APPROVAL:
The study was approved by the Institutional Ethical Review Committee (ERC) bearing the registration number 2023- 9134-26167.

COMPETING INTEREST:
The authors declared no conflict of interest.

AUTHORS’ CONTRIBUTION:
SMRK: Conceptualised this research.
SMRK, FAS: Collected and analysed the data and prepared the manuscript.
TZ: Critically reviewed the manuscript.
All authors approved the final version of the manuscript to be published.

REFERENCES

1. Chimea T, Kanji Z, Schmitz S. Assessment of clinical competence in competency-based education. Can J Dent Hyg 2020; 54(2):83-91.
2. Willis BW, Campbell AS, Sayers SP, Gibson K. Integrated clinical experience with concurrent problem-based learning is associated with increased clinical reasoning of physical therapy students in the United States. J Educ Eval Health Prof 2018; 15:30. doi: 10.3352/jeehp.2018.15.30.
3. Lee GB, Chiu AM. Assessment and feedback methods in competency-based medical education. Ann Allergy Asthma Immunol 2022; 128(3):256-62. doi: 10.1016/j.anai.2021.12. 010.
4. Slavec A, Drnovsek M. A perspective on scale development in entrepreneurship research. Economic Business Rev 2012; 14(1). doi: 10.15458/2335-4216.1203.
5. Graham R, Zubiaurre Bitzer LA, Mensah FM, Anderson OR. Dental student perceptions of the educational value of a comprehensive, multidisciplinary OSCE. J Dent Educ 2014; 78(5):694-702. doi: 10.1002/j.0022-0337.2014.78.5.tb 05721.x.
6. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommen-dations. Res Nurs Health 2007; 30(4):459-67. doi: 10.1002/ nur.20199.