|Year : 2020 | Volume
| Issue : 7 | Page : 3399-3404
Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education
Alka Rawekar1, Sonali G Choudhari2, Vedprakash Mishra3, Sunita Vagha4
1 Department of Physiology, Jawaharlal Nehru Medical College; Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India
2 Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences; Department of Community Medicine, Jawaharlal Nehru Medical College, Wardha (MS), Maharashtra, India
3 Pro Chancellor, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India
4 Ex-Director, School for Health Professions Education and Research; Department of Pathology, Jawaharlal Nehru Medical College, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India
|Date of Submission||30-Jan-2020|
|Date of Decision||13-Mar-2020|
|Date of Acceptance||07-Apr-2020|
|Date of Web Publication||30-Jul-2020|
Dr. Sonali G Choudhari
Professor, Department of Community Medicine, Jawaharlal Nehru Medical College, Head, Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences, Wardha, Maharashtra
Source of Support: None, Conflict of Interest: None
Background: Observational assessment (OA) or Workplace Based Assessment (WPBA) is being increasingly used to assess the trainees by direct observation to shape their learning. The current deficiencies in our conventional assessment system are due to lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools of assessment. Objective: The present study aimed to introduce formative assessment (FA) for postgraduates at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (Deemed University) Sawangi (M) Wardha, (Maharashtra), India. Methodology: Observational Assessment tools used for FA were Mini CEX (Mini Clinical Evaluation Exercise) and DOPS (Direct Observation of Procedural Skills)in addition to Objectively Structured Clinical/Practical Examination (OSCE/OSPE). Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (Preclinical, Para-clinical, Clinical). Results: A significant improvement in scores from one semester to other as the postgraduate learner progresses, was observed. The overall response from the postgraduate students and faculty for FA was satisfactory. Conclusion: Formative assessment in practical's, was feasible, acceptable and effective program to improve the overall learning and competency of postgraduates.
Keywords: Formative assessment, observational assessment, postgraduates
|How to cite this article:|
Rawekar A, Choudhari SG, Mishra V, Vagha S. Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education. J Family Med Prim Care 2020;9:3399-404
|How to cite this URL:|
Rawekar A, Choudhari SG, Mishra V, Vagha S. Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education. J Family Med Prim Care [serial online] 2020 [cited 2020 Aug 13];9:3399-404. Available from: http://www.jfmpc.com/text.asp?2020/9/7/3399/290765
| Background|| |
“Assessments drives students’ learning” is the accepted quote. But what is rather strange is how assessment directs, drives, and influence learning. This understanding holds the key for an educator, teacher, or policymaker to direct the entire process of learning through appropriate assessment tools.
One of the models of competency, the Miller's Pyramid, provides a basic framework for bringing into focus the four levels of competence: ‘Knows', ‘Knows how', ‘Shows how', and ‘Does’ (Miller, 1990). Multiple choice questions or short answered questions or long answered questions can test the first two levels while the latter two components can be tested by objective structured practical/clinical examination (OSPE/OSCE) and other performance-based assessment tools.
In India, the Postgraduate Medical Education Regulations 2000 of the Medical Council of India (MCI) states that postgraduate training be competency-based and also suggests the use of ‘Logbook’ for monitoring the learning process. However, these regulations do not provide any process or road map of in-training assessments. There are few gray areas in the document of Graduate Medical Regulations of MCI 1997 like ‘integrated teaching', which has got somewhat disappointing results due to lack of provision of operational details or/and monitoring details. While there is a provision of internal assessment for periodic assessments during the undergraduate (UG) program in Indian Medical Regulations, there are no such pre-requisites for postgraduate (PG) courses in India.
PG training is directed not merely at the attainment of knowledge, attitude, and skills but also observable responsiveness and appropriate functioning in real-life situations. The purpose of PG education is to create specialists who would provide high- quality healthcare and advance the cause of science through research and training. At the same time, they would also serve as physicians of the first contact for the community, practicing affordable, accessible primary healthcare to the rural, needy, marginalized, and underprivileged populace.
It follows that even the most ideal of conventional assessments conducted in examination settings will fall short of measuring these outcomes. There is indeed a need to observe and assess the trainees in real situations so that necessary corrections can be provided to the trainees. Observational assessment (OA) or Workplace-based assessment (WPBA) is being increasingly used to assess the trainees by direct observation and to shape their learning., The current deficiencies in the assessment system of India are due to a lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools. Hence, considering the Indian scenario, using the advantages of OA over the traditional assessment methods and its educational utility; the study aimed to introduce formative assessment (FA) in practicals for PGs at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (DU), Wardha (MS), India. The objectives of the study were to train and sensitize the faculty/supervisors and PG trainees for FA in PG practical as “Assessment for Learning” and to evaluate the program for its feasibility and effectiveness.
| Materials and Methods|| |
The study was conducted from June 2015 to October 2016, in all the Departments (having Postgraduate courses) of Medical, Dental, Ayurveda, and Nursing Colleges, of DMIMS (DU), Wardha (MS), India. The study participants were the PG students of the first and second academic year and faculty/guides/supervisors of the respective departments. The approval from Institutional Ethics Committee was obtained (Ref No.: DMIMS (DU)/IEC/2015-16/155).
It was an interventional prospective analytical study. OA tools used for FA were mini clinical evaluation exercise (MiniCEX) and direct observation of procedural skills (DOPS) in addition to OSCE/OSPE. Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (preclinical, paraclinical, clinical). There was the flexibility of the sites for using these tools as outpatient department (OPD) setting, in-patient department (IPD) settings, community/fieldwork, operation theater etc., as per the requirement of the subject/department.
Sensitization of the faculty was done for the concept of ‘FA in practical for postgraduate'. The faculty was trained for using the tools for OA, i.e., MiniCEX and DOPS as well as in providing developmental feedback on direct observation. The training was given by conducting a one-day workshop for the faculty. Similarly, the PG students were also sensitized by the respective Heads of the Department regarding the importance and implementation strategy of FA in practical.
All the departments identified various competencies and skills to be attended by the junior residents and prepared checklists for OA as well as various stations for OSPE/OSCE. Procedural skills were identified as per the level of residency of the PG students. All the skills or competencies identified by subject experts to be used for MiniCEX or DOPS and OSCE/OSPE stations were prevalidated by the respective Heads of the Department followed by rigorous validation by faculties of the Department of Assessment and Evaluation, School for Health Professions Education and Research.
The scores assigned for each FA (per semester) in practical for PGs were as follows:
- 3 OAs at the end of first semester = 30 marks
- 3 OAs at the end of second semester = 30 marks
- OSPE/OSCE examination (8 stations) =40 marks
(Total of 100 marks for each academic year).
Later, as per the operational strategy for FA in practical, validated tools for identified skills were implemented for FA in practical at the end of each semester. The result was prepared by the respective Department and submitted for final analysis. Feedback from PG teachers/supervisors and PG students were solicited for the feasibility, practicality, and applicability of FA in practical for PGs in the form of the validated structured feedback proforma. All the enrolled PGs (n = 331) and all registered PG teachers/supervisors (n = 153) participated in the study.
| Results|| |
Analysis of marks/scores obtained in Formative assessment
[Table 1] depicts the mean scores of the FA (OA and OSPE/OSCE) of the PGs of various institutes under the ambit of the university. The mean scores of Junior Resident 1 (JR1) and Junior Resident 2 (JR2) are calculated for the FA. Overall,
|Table 1: Institution wise distribution of marks of Formative assessment (Observational assessment and OSPE/OSCE) of the postgraduates|
Click here to view
- The mean scores of FA ranged in between 58.11 and 64.18 out of 100 marks
- The highest score of 64.18 marks out of 100 is of the JR2 of medical college
- For the institute of Ayurveda, the mean FA score for JR1 was 50.21/80 marks and for JR2 it was 75.83/120 Marks
- In all the institutes, for both JR1 and JR2, mean scores of 1st OA were improved in the 2nd OA test
- The mean scores of OSPE/OSCE were also above 50% for both JR1 and JR2 in all the institutes.
Line diagrams [Figure 1] depicting the trend of the mean scores of OA for JR1 of Medical college showed improvement in scores.
|Figure 1: Trend of the scores of observational assessment for JR1 of Medical college|
Click here to view
Out of all the departments of medical faculty, having PG students (JR1) of the Batch 2015-16, in 4 departments (ENT, Medicine, Pediatrics, Radiology) the mean FA score came below 50%. Similarly, for JR2 of the batch 2014-15, in one department (Community Medicine) the mean FA score was found to be below 50% [Figure 2] and [Figure 3].
|Figure 2: Department-wise distribution of mean scores of formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR1) of Medical college|
Click here to view
|Figure 3: Department wise distribution of mean scores of Formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR2) of Medical college|
Click here to view
On similar lines, out of all the departments of dental faculty having PG students (JR1) of the Batch 2015-16, in only one department (Oral Medicine & Radiology) the mean FA score came below 50%. For JR2, all the departments of dental faculty had the mean FA score above 50% [Figure 4].
Total FA score of all departments of Ayurveda faculty for JR1 is calculated considering one OA test and OSPE/OSCE marks (as the first OA was not conducted due to late admissions. See Table No. 1). The total marks are out of 80 for JR1 of Ayurveda. For JR1, all the departments of Ayurveda faculty had the mean formative assessment score above 50%. All the departments of Ayurveda College, having PG students (JR2) of the Batch 2014-15, the mean FA score was above 50% (>60 marks).
For the faculty of Nursing, in all the departments, the JR1 scored above 50% for the FA.
Feedback analysis for Formative assessment
The response rate of feedback from all PG students was 92% whereas from PG teachers/supervisors of all the constituent colleges was 79%. For almost all the parameters of feedback, the PG students showed satisfaction towards the various aspects of FA. The rating average ranged between 3.86 and 4.19 [Figure 5]. Overall rating by the PGs and faculty for the process of FA showed that cumulatively 88.27% PGs and faculty rated the FA as Good (72.22%) to Best (16.05%).
|Figure 5: ‘Rating average’ for feedback from postgraduate students and faculty regarding FA (measured on 5 point Likert scale)|
Click here to view
Key suggestions to improve Formative assessment in the department given by students and teachers
- FA provides a good opportunity for PGs to self-assess their practical knowledge
- Should be undertaken every fortnightly or monthly
- Helps to improve the PG's practical and communication skills with patients
- Should be included in summative assessment too
- OSCE should be done in both IPD and OPD settings
- Combination of OSCE, MiniCEX/DOPS should be applied to every patient's procedure (minor surgical) for the betterment of results
- More procedures to be covered through DOPS/MiniCEX
- It should not only be taken for exam purpose but also taken once in a week to improve patient examination skills
- Should be implemented as a routine day to day system (such as during rounds or immediate after wand rounds)
- Supervised learning may be useful
- For improvement in FA, only procedural skills should not be chosen for DOPS but it should include problem-solving skills in the form of case scenario and differential diagnosis, clinical application of the procedure, etc
- Repeated conduct should be monitored for improvement in outcome.
Hindrances faced by postgraduate students and faculty during the implementation of Formative assessment in the department
Majority of the PG students opined that the FA was beneficial and there were no significant hindrances during its implementation.
Some of the hindrances informed by others were as follows: -
- Time constraint, Space constraint, in some departments
- If it is conducted in routine clinical working hours, there is a rush by both PGs and faculty to get back to the routine clinical duties. Hence, FA should be taken in a separate time slot
- Need to have many more faculty/assessors available in the department on the day of FA, in major clinical departments with a significant number of PGs
- It is tedious to prepare a checklist for each and every procedure. If there is no trained staff for how to prepare the checklist or how to conduct DOPS, then responsibility lies with only trained staff in the department
- More time is required for doing all things meticulously. Because of the task of completing clinical and/or departmental work, faculty may not give much time for everyone during FA.
| Discussion|| |
Health educators have differentiated assessment according to its purpose; assessment for learning (FA) and assessment of learning (Summative assessment). When the information from an assessment is used solely to make a judgment about the level of competence or achievement, it is the summative assessment. FA refers to assessment that is particularly intended to provide feedback on performance to improve and augment learning. It is not the instrument that is formative, it is the use of the information gathered, by whatever means, to adjust teaching and learning that merits the “formative” label.
Post Graduate Medical Education Regulation 2000 document of the Medical Council of India (the regulatory body of Medical Education in India) mentions that the PG curriculum should be ‘competency-based'. It should ideally develop competent, confident, concerned, compassionate, and globally relevant healthcare professionals who must provide comprehensive healthcare. However, there is a need for specific guidelines regarding assessing those competencies. At present, the three- year long PG training is assessed at the endpoint only (summative assessment). This means that the assessment is done for the product and not for the process. There are very few colleges/Universities conducting FA in theory for PG in India. But very little literature is available for FA especially for assessing competency through practical, for all health professions.
DMIMS (DU), Wardha is a Deemed University (in Rural setting) having Medical, Dental, Ayurved, and Nursing Colleges, which are regulated by their respective Apical Councils. These councils do not encompass any form of formative assessment or continuous internal assessment in their PG training program. Knowing the importance of FA for learning, we conducted a study for PG students during the academic year 2015-16 for practical (in the domain of skills and attitude) and tested for its feasibility and effectivity.
In this study, it was observed that the mean score of OA conducted for the second time was improved as compared to the first OA test scores. Though these scores were not statistically significant the increasing trend was observed for both JR1 and JR2. OSPE/OSCE was one of the modalities of the FA. OSCE was used as a tool for both formative and summative assessment of Japanese PG in clinical training at Hiroshima, wherein it was found that trainees viewed OSCEs positively and appreciated their effectiveness from a pedagogical perspective, and OSCE positively affected the trainees’ approach to learning. Similarly, Miller JK, in his study, on ‘Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy', an adaptation of the OSCE procedures was done for competency-based training of Marriage & Family Therapy students. Instead of using the procedures as a summative examination as is typical in medical education, the study proposed how to use them as formative exercises in the development of student competence.
The study by Dijksterhuis M G et al. aimed to explore perceptions of PG and teachers about factors that determine active engagement in FA. It was stated that engaging in FA with a genuine impact on learning is complex and quite a challenge to both students and teachers. Increased acceptability along with the effective implementation of FA structure, individual perspectives on feedback, a supportive learning environment, and credibility of feedback are all important in this process. Lases LSS et al. (2019) too opined that a healthy learning environment is associated with improved resident well-being. Every one of these should be taken into account when the utility of FA in PG medical training is evaluated. On the other hand, as per Watling and Ginsburg (2019), the emergence of learning from a cauldron of assessment and feedback can seem like alchemy. Whereas feedback stresses development and learning, assessment stresses judgment and decision making. Mixing these distinct ingredients creates a risky recipe for learning.
In this study, all the PG residents and teachers/assessors were asked to give feedback regarding the feasibility and utility of this program in a structured questionnaire. Feedback analysis depicted the acceptance of FA by majority of students and faculties.
| Conclusions|| |
This study of ‘Formative Assessment in Practical for postgraduates in Health profession at DMIMS (DU) Wardha (MS), India', revealed that it was a feasible, acceptable, and effective program (with minor modifications at respective departments) to improve the overall learning of students. From the results, it can be concluded that the overall response from the PG students and faculty for FA was good. The results showed a significant improvement in scores from one semester to other as the PG learner progressed.
As it was a pilot study conducted only for 1 year, the prospective progress of the PG student could not be computed.
Looking at the need for FA for Learning in Health Education, the following recommendations are made.
- To include OSCE/OSPE in summative assessment of PG training
- Increasing the number of OAs to have more reliability and thus increasing validity
- Incorporation of FA (practical) in PG training program as ELIGIBILITY CRITERIA (minimum 35% or 50%) for progression
- Inclusion of the concept of Continuous Internal Assessment in Post Graduate Assessment, as in the case of undergraduates
- It should be included in the curriculum as a part of the scheme of examination
- Bank of OSCE station for PG should be prepared at the departmental level.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Adkoli BV. Assessment of professionalism and Ethics. In: Singh T, Anshu, editor. 1st
ed. Jaypee Brothers Medical Publishers: New Delhi; 2012. p. 180-190.
Miller G. The assessment to clinical skills, competence, performance. Acad Med 65;1990:S63-7.
Medical Council of India. Postgraduate Medical Education Regulations, 2000. Available from: http://www.mciindia.org
. [Last accessed on 2015 Mar 11].
Medical Council of India. Graduate Medical Regulations, 1997. Available from: http://www.mciindia.org
. [Last accessed on 2015 Mar 11].
Norcini JJ, Burch V. Workplace based assessment as an educational tool: AMEE guide No. 31. Med Teach 2007;29:855-71.
Downing SM, Yudkowsky R. Observational Assessment; Goals and Tools. Assessment in Health Professions Education Routledge Publication, New York. First Ed 2009.
Van der Vleuten CPM, Schuwrith CWT. Assessing professional competence from methods to program. Med Educ 2005;39:309-17.
Medical Council of India Postgraduate Medical Education Regulations, 2000, (amended upto January 2017), Medical Council of India Pocket – 14, Sector 8, Dwarka, NEW DELHI – 110 077. Available from: http://www.mciindia.org
Taguchi N, Ogawa T. OSCEs in Japanese postgraduate clinical training Hiroshima experience 2000-2009. Eur J Dent Educ 2010;14:203-9.
Miller JK. Competency-based training: Objective structured clinical exercises (OSCE) in marriage and family therapy. J Marital Fam Ther 2010;36:320-32.
Dijksterhuis MG, Schuwirth LW, Braat DD, Teunissen PW, Scheele F. A qualitative study on trainees’ and supervisors’ perceptions of assessment for learning in postgraduate medical education. Med Teach 2013;35:e1396-402.
Lases LSS, Arah OA, Busch ORC, Heineman MJ, Lombarts K. Learning climate positively influences residents’ work-related well-being. Adv Health Sci Educ Theory Pract 2019;24:317-30.
Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ 2019;53:76-85.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5]