|Year : 2021 | Volume
| Issue : 3 | Page : 464-468
Strengths, weaknesses, and suggestions for improvement in postgraduate assessment in community medicine in India: A Delphi study
Amol R Dongre1, John Norcini2
1 Department of Extension Programmes (SPARSH), Pramukhswami Medical College (PSMC), Karamsad, India
2 SUNY Upstate Medical University, Syracuse, New York, USA
|Date of Submission||08-Sep-2020|
|Date of Acceptance||21-Apr-2027|
|Date of Web Publication||13-Oct-2021|
Dr. Amol R Dongre
Department of Extension Programme (SPARSH), Pramukhswami Medical College, Karamsad, Gujarat
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Objectives: It was to understand the strengths and weaknesses of the current postgraduate assessment system in community medicine in India, to identify recommendations for change, and to build a consensus around them. Materials and Methods: A conventional Delphi technique was preferred for consensus building among experts. We completed three Delphi rounds over a period of 4 weeks, and 16 experts participated in the study. Content analysis was done for open-ended responses, and consensus analysis was done for Likert-type scale questionnaire. In round three, we obtained their top five preferences for change in assessment. Results: The experts agreed to have an assessment system based on ongoing formative and one end-of-year summative assessment. Apart from this, they agreed on the various occasions for carrying out the formative assessment. Furthermore, they clearly agreed on measures such as blueprinting, improving test formats, and adequate briefing of test-taking students. Conclusion and Recommendations: Most of the consensus items were found to be in alignment with the modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to enhance its validity and reliability.
Keywords: Assessment, community medicine, India, postgraduate
|How to cite this article:|
Dongre AR, Norcini J. Strengths, weaknesses, and suggestions for improvement in postgraduate assessment in community medicine in India: A Delphi study. Indian J Community Med 2021;46:464-8
|How to cite this URL:|
Dongre AR, Norcini J. Strengths, weaknesses, and suggestions for improvement in postgraduate assessment in community medicine in India: A Delphi study. Indian J Community Med [serial online] 2021 [cited 2021 Nov 27];46:464-8. Available from: https://www.ijcm.org.in/text.asp?2021/46/3/464/328194
| Introduction|| |
The Lancet Commission on the Education of Health Professionals for the 21st century, highlighted that the present curriculum in health profession education is fragmented and outdated and recommended reform in health profession education and assessment to supply skilled professionals to health-care systems. In India, the Medical Council of India has envisioned reforms in the postgraduate curriculum of all the subjects. In recent times, there has been explicit discussion of improving training and assessment in postgraduate courses in the overall subjects, pharmacology, and Ophthalmology. Nevertheless, the National Medical Commission envisions reform in all postgraduate programs including community medicine.
Over the years, postgraduates in community medicine have been assessed using a traditional assessment framework. The decision about final certification highly relies on the student's performance in a single final university examination. This assessment framework is less defensible as it uses fewer methods which are focused on knowledge assessment and done on one occasion in the course. Although teachers and experts feel the need for a change and improvement in the current assessment practices,, there is limited systematic effort on development of consensus on what to improve and how to undertake the change? with the group of experts. Hence, the aim of the present study was to understand the strengths and weaknesses of the overall current assessment system in community medicine in India and to build a consensus on recommendations for improvement in the overall assessment.
| Materials and Methods|| |
Research design and assumptions
The present study was based on the constructivist paradigm of knowledge building in research. Being a complex issue, assessment reform requires discussion and accommodation of different views. A conventional Delphi technique was preferred for the present study as it allows iterative data collection for exploration and consensus building from a panel of experts who are geographically scattered. The Delphi method assumes that group opinions are more valid than individual judgments.
Sample size, sampling, and access to sample
There is no agreement on the number of experts required for the Delphi technique, however, it has been found that a panel of 10–15 experts is usually enough to explore the topic and build a consensus. Out of 18 experts invited, 16 experts in the field of community medicine from India participated in the present study. The expert was defined to be an individual with relevant knowledge and experience on a particular topic or the one who has a position in the hierarchy, a recognized person in the field, or someone recommended by other participants.
We used criterion type of purposive sample of experts who were accessible, willing to express, experienced in postgraduate assessment, and had training in the field of medical education. The purpose was to capture their wisdom acquired during professional practice. The experts had different levels of hierarchy, affiliations to type of university, and geographical regions. The details of their characteristics are given in the first paragraph of results section.
Study tools, data collection, and analysis procedures for each Delphi round
The study was proved by SMVMCH-Institutional Ethics Committees (EC code No: 14/2018).
Delphi round one: Data collection and analysis
In the first round of Delphi, experts (n = 18) were asked to respond to a questionnaire consisting of broad, open-ended questions based on the research questions. The purpose of this round was to explore the strengths, weaknesses, and suggestions for improvement in assessment from the experts' point of view and finally develop the questionnaire for round two.
We used electronic mail (e-mail) to interact with the Delphi experts. This virtual mode of communication was found to be cheap, rapid and was found to yield high response rate. The soft copy of the predesigned and pretested questionnaire was E-mailed to their preferred E-mail address; they were given 2 weeks to respond.
It is to be noted that researchers' academic background, and their pre-conceived ideas about assessment might have influenced the research process and interpretations. It involves summarization and classification of data as categories and themes in simple quantitate terms. We used a practical approach to verify the categories generated, where the first investigator (AD) carried out the categorization with his other colleague in the department. Both were trained in the use of qualitative research methods and data analysis. Later, the second author (JN) reviewed the complied findings for face validity.
Delphi rounds two and three: Data collection and analysis
In the second round, Delphi experts (n = 16) were E-mailed a soft copy of a Likert-type scale based on findings from round one. The experts were asked to rate the 14 items on a five-point Likert-type scale having options from “strongly agree to strongly disagree.” Experts were asked to consider feasibility, existing resources, and context while rating the items.
In the third round of Delphi, the experts (n = 16) were asked to select their top five among the items which reached consensus of more than 90% in the second round. One of the experts refused to rank order as he felt that all items were relevant and mutually related. The experts were also given an opportunity to revise their opinions on the items which did not achieve consensus in the second round of Delphi. All three rounds were completed over 4 weeks.
We used consensus measure to analyze experts' rating on items on suggestions for improvement in the assessment practices. We calculated weighted mean (WM) and the consensus values which range from 0% to 100%, where 0 indicates no consensus at all and 100% indicates full consensus. A value above 80% was considered to reflect good consensus. In round three, experts were asked to select their top five items among items which reached a consensus of more than 90% in round two. The purpose was to find out the most prominent and representative suggestions for the improvement in assessment by calculating Smith's salience score, where score close to one indicates the prominence of items across the lists.
| Results|| |
We had a high response rate (89% for round one and 100% for rounds two and three). Out of 16 experts, 9 (56.2%) were male and 7 (43.8%) were female. Half of the experts were affiliated with government medical colleges, 6 (37.5%) were working in private medical colleges, and 2 (6.2%) experts were affiliated with a semi-government medical college in India. Out of 16, two experts had their medical college affiliated with a deemed university, and 14 (87.5%) experts had their college affiliated with the government universities. Six experts were professor and head, five were professors, two were associate professor, and one each was additional professor, dean (medical education), and a consultant to the university on curriculum development. All of them had interest and training in the field of medical education.
Results from Delphi round one
Overall, experts were concerned about the weaknesses in the current assessment system. Six experts mentioned that it is highly focused on testing the cognitive domain with little emphasis on competence or skills development. Two experts had concerns about the unfair and subjective nature of the current system as the pass/fail decision relies on performance in only one final test. To improve the overall assessment system, the suggestions varied from a radical change to incremental modifications to the current system. Four experts suggested breaking the course into smaller learning units with summative assessment at the end of each unit and formative assessments between the summative tests. They felt that the assessment system should be based on an assessment of competencies and a blueprint should bridge the gap between them and the assessments. The experts emphasized the need for formative assessment, choice of tools, selection, and training of examiners and development of an evaluation system for the overall and incremental improvement in the system [Table 1].
|Table 1: Experts' opinion on strengths, weaknesses, and suggestions for improvement in overall assessment system (n=16)|
Click here to view
Results from Delphi round two
Consensus on suggestions for improvement in overall assessment
Out of 14 items, there was a good consensus (score >80%) on eight items. There was a consensus for aligning the assessment with the desired competencies (100%) and developing the formative assessment system (WM – 1.1; score – 91.5%). The experts agreed on the need to have a blueprint to align test content with the desired competencies (WM – 1.1; score – 91.5%) and they agreed that the blueprint should be shared with all the examiners (WM – 1.1; score – 91.5%). Experts thought that the examiners should be trained in the assessment tools which they use (WM – 1.3; score – 83.4%). Notably, they agreed to have an exit feedback system in place to obtain students' feedback on their experiences with current assessments (WM – 1.5; score – 80.7%) [Table 2].
|Table 2: Average rank of the ten items with the highest consensus (>90%) (n=15)|
Click here to view
Results from Delphi round three
The five most salient (prominent) items among the among the ten highest consensus (>90%) points were related to the need for having a competency-based assessment (Smith's S – 0.85), the need for formative assessment (0.43), the need for occasions for formative assessment (0.37), and the revision of the current logbook to include students' reflections (0.31) and structured guidelines for formative assessment (0.27) [Table 3].
|Table 3: Model framework for postgraduate assessment based on expert consensus|
Click here to view
| Discussion|| |
The panel of experts had a mixed reaction to the current assessment system. Some felt that the current assessment system is fair, time-tested, and appropriate for the resources and context, while other experts pointed out that it is prejudicial, subjective, and focused too much on the assessment of lower-order cognitive domains of Bloom's taxonomy such as remembering and understanding. Notably, postgraduate assessment in other subjects of medicine in India and overall, in the South Asian region is also based on lower-order knowledge assessment.
The experts agreed to have an assessment system of community medicine based on ongoing formative and one end-of-year summative assessment. They emphasized the need to align the desired competencies and assessment measures. The experts agreed to the occasions such as community posting at rural health training center/urban health training center/primary health centers/posting at national health programs for formative assessment. In the West, such occasions are used for workplace-based assessment. The success of such assessments depends on the student–teacher interactions, adequate number of assessments using a valid and reliable tool, and faculty training.
In community medicine, the practice of maintaining PG logbooks is almost universal and it primarily includes accounts of activities done at different sites by the postgraduates. The experts agreed to improve the scope of exiting PG logbook by providing space for student's reflections on the activities with regular faculty supervision. The key practical tip includes aligning the learning objectives of the logbook with the curriculum; these learning objectives should be the part of the assessment.
The expert panel suggested a better use of the current measures by bringing more objectivity to scoring through the introduction of guidelines and checklists. Overall, the experts clearly agreed that measures such as blueprinting, improving test formats, adequate briefing of test-taking students to reduce their anxiety, the training of examiners, and better test conditions would improve the validity and reliability of assessment scores. It is noteworthy that most of the consensus items in the second round of Delphi meet the requirements for the development of an effective assessment system. It is encouraging to note that the experts achieved consensus on having an exit feedback system in place to obtain students' views on their recent assessments, which is expected to help in an evaluation of overall assessment system.
The summative assessment which is conducted by the university as an external examination includes both written tests and practical skills. In India, medical teachers have limited training in the “correct” use of traditional assessment measures and there is limited value given to feedback. Since it is the “users” who can make a difference, formal training of teachers and their behavior change would be the key to successful transformation. Therefore, experts suggested a better use of traditional assessment measures through faculty training and support. There are programs where a mix of formative and summative assessment, blueprinting, and training of faculty in the use of assessment measures could achieve its key goals.
Experts could not reach a clear consensus on having standards for external examiners. “An external examiner is defined as a visiting assessor of high academic standing and possessing absolute integrity and objectivity – appeals to the notions of universality of educational standards and justice for the individual student” (pp. 1093). In the United Kingdom, external examiners and in North America, external examinations have a role in ensuring quality in higher education. More discussion and research are required on this topic.
There was no consensus on the minimum pass level (i.e., standard setting). According to Postgraduate Medical Education Regulation in India, a student is required to score 50% in written test and 50% in practical skills assessments in their end-of-course summative examinations. This is conventional arbitrary cutoff value. However, there is a growing realization that the pass–fail cutoffs should be based on scientific standard setting procedures to make them more credible and defensible. There are several standard setting methods, and none is agreed upon as the best method, hence the choice depends on a method that is acceptable in the given context. [Table 3] summarizes the model framework for assessment based on findings.
To the best of our knowledge, the present Delphi study is the first systematic effort on this topic. The high response rate and rigorous scientific data collection and analysis brought credibility to the study findings. The limitations of the present study should be kept in mind. The researchers' own academic background in the field of community medicine and their qualification in the field of medical education might have influenced the research design and its results. Study participants' prior training in the field of medical education might have influenced their responses in support of modern assessment practices. However, the findings of the present study can be used for a large-scale survey.
| Conclusion|| |
In summary, most of the consensus items were in alignment with modern assessment theory. Regulating body and policymakers should revise the current postgraduate assessment system in community medicine to standardize and enhance its validity and reliability of assessment to produce competent professionals for health-care system.
This article is based on a dissertation titled “Strengths, weaknesses and suggestions for improvement in postgraduate assessment in Community Medicine in India: A Delphi study” for the degree of Master of Health Professions Education by Amol Dongre guided by Professor John Norcini, submitted to Keele University, UK, in the year 2018 under FAIMER-Keele Distance Learning Master's Course. We thank all the Delphi experts for their contribution.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al.
Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet 2010;376:1923-58.
Mahajan R, Anshu, Gupta P, Singh T. Practice-based learning and improvement (PBLI) in Postgraduate Medical Training: Milestones, instructional and assessment strategies. Indian Pediatr 2017;54:311-8.
Badyal DK, Desai C, Tripathi SK, Dhaneria SP, Chandy SJ, Bezbaruah BK. Postgraduate pharmacology curriculum in medical institutions in India: Time for need-based appraisal and modifications. Indian J Pharmacol 2014;46:584-9.
] [Full text]
Ajay K, Krishnaprasad R, Divya DS. Ophthalmic surgical training in Karnataka and Southern India: Present status and future interests from a survey of final-year residents. Indian J Ophthalmol 2015;63:306-11.
] [Full text]
Ananthakrishnan N, Arora NK, Chandy G, Gitanjali B, Sood R, Supe A, et al.
Is there need for a transformational change to overcome the current problems with postgraduate medical education in India? Natl Med J India 2012;25:101-8.
Sood R, Singh T. Assessment in medical education: Evolving perspectives and contemporary trends. Natl Med J India 2012;25:357-64.
Creswell JW, Vicki L, Clark P. Designing and Conducting Mixed Methods Research. 2nd
ed. New Delhi: Sage Publications; 2011.
Abramson JH, Abramson ZH. Survey Methods in Community Medicine. 4th
ed. New York: Churchill Livingstone; 1997.
Habibi A, Sarafrazi A, Izadyar S. Delphi technique: Theoretical framework in qualitative research. Int J Eng Sci 2014;3:8-13.
Fuller M, Henderson S, Bustamante R. Assessment leaders' perspectives of institutional cultures of assessment: A Delphi study. Assess Eval High Educ 2015;40:331-51.
Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract 1996;4:67-74.
Gessaroli M, McKinley D. Research Designs and Sampling: FAIMER-Keele Master's in Health Professions Education: Accreditation and Assessment. Module 6, Unit 3. 3rd
ed. London: Keele University and CenMEDIC; 2017.
Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate research. J Inform Technol Educ 2007;6:1-21.
Silverman D. Interpreting Qualitative Data: A Guide to the Principles of Qualitative Research. London: SAGE Publications; 2011.
Gibbs G. Analytic Quality and Ethics. London: SAGE Publications; 2007.
Tastle WJ, Russell J, Wierman MJ. A new measure to analyse student performance using the Likert scale. Inform Sys Educ J 2008;6:1-9.
Bernard HR, Wutich A, Ryan GW. Analysing Qualitative Data: Systematic Approaches. 2nd
ed. New Delhi: SAGE Publications; 2017.
Mendis L, Adkoli BV, Adhikari RK, Huq MM, Qureshi AF. Postgraduate medical education in South Asia: Time to move on from the postcolonial era. Br Med J 2004;328:779.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-71.
Schüttpelz-Brauns K, Narciss E, Schneyinck C, Böhme K, Brüstle P, Mau-Holzmann U, et al.
Twelve tips for successfully implementing logbooks in clinical training. Med Teach 2016;38:564-9.
Norcini JJ. Current perspectives in assessment: The assessment of performance at work. Med Educ 2005;39:880-9.
West R, Crighton J. Examination reform in Central and Eastern Europe: Issues and Trends. Assess Educ 1999;6:271-80.
Sharma S, Sharma V, Sharma M, Awasthi B, Chaudhary S. Formative assessment in postgraduate medical education – Perceptions of students and teachers. Int J Appl Basic Med Res 2015;5:S66-70.
Vleuten CV, Verhoeven B. In training assessment development in postgraduate education in Europe. ANZ J Surg 2013;83:454-9.
Smith JD, Prideaux D, Wolfe CL, Wilkinson TJ, Sen Gupta T, DeWitt DE, et al.
Developing the accredited postgraduate assessment program for Fellowship of the Australian College of Rural and Remote Medicine. Rural Remote Health 2007;7:805.
Walters WA, Sivanesaratnam V, Hamilton JD. External examiners. Lancet 1995;345:1093-5.
Campbell PN. Some thoughts on the role of external examiners. Biochem Educ 1991;19:13.
Mortaz Hejri S, Jalili M. Standard setting in medical education: Fundamental concepts and emerging challenges. Med J Islam Repub Iran 2014;28:34.
Barman A. Standard setting in student assessment: Is a defensible method yet to come? Ann Acad Med Singap 2008;37:957-63.
[Table 1], [Table 2], [Table 3]