Home Print this page Email this page Small font size Default font size Increase font size
Users Online: 4722
Home About us Editorial board Search Ahead of print Current issue Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents 
ORIGINAL ARTICLE
Year : 2020  |  Volume : 9  |  Issue : 7  |  Page : 3399-3404  

Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education


1 Department of Physiology, Jawaharlal Nehru Medical College; Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India
2 Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences; Department of Community Medicine, Jawaharlal Nehru Medical College, Wardha (MS), Maharashtra, India
3 Pro Chancellor, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India
4 Ex-Director, School for Health Professions Education and Research; Department of Pathology, Jawaharlal Nehru Medical College, Datta Meghe Institute of Medical Sciences, Wardha (MS), Maharashtra, India

Date of Submission30-Jan-2020
Date of Decision13-Mar-2020
Date of Acceptance07-Apr-2020
Date of Web Publication30-Jul-2020

Correspondence Address:
Dr. Sonali G Choudhari
Professor, Department of Community Medicine, Jawaharlal Nehru Medical College, Head, Department of Assessment and Evaluation, School for Health Professions Education and Research, Datta Meghe Institute of Medical Sciences, Wardha, Maharashtra
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jfmpc.jfmpc_185_20

Rights and Permissions
  Abstract 


Background: Observational assessment (OA) or Workplace Based Assessment (WPBA) is being increasingly used to assess the trainees by direct observation to shape their learning. The current deficiencies in our conventional assessment system are due to lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools of assessment. Objective: The present study aimed to introduce formative assessment (FA) for postgraduates at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (Deemed University) Sawangi (M) Wardha, (Maharashtra), India. Methodology: Observational Assessment tools used for FA were Mini CEX (Mini Clinical Evaluation Exercise) and DOPS (Direct Observation of Procedural Skills)in addition to Objectively Structured Clinical/Practical Examination (OSCE/OSPE). Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (Preclinical, Para-clinical, Clinical). Results: A significant improvement in scores from one semester to other as the postgraduate learner progresses, was observed. The overall response from the postgraduate students and faculty for FA was satisfactory. Conclusion: Formative assessment in practical's, was feasible, acceptable and effective program to improve the overall learning and competency of postgraduates.

Keywords: Formative assessment, observational assessment, postgraduates


How to cite this article:
Rawekar A, Choudhari SG, Mishra V, Vagha S. Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education. J Family Med Prim Care 2020;9:3399-404

How to cite this URL:
Rawekar A, Choudhari SG, Mishra V, Vagha S. Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education. J Family Med Prim Care [serial online] 2020 [cited 2020 Aug 14];9:3399-404. Available from: http://www.jfmpc.com/text.asp?2020/9/7/3399/290765




  Background Top


“Assessments drives students’ learning” is the accepted quote. But what is rather strange is how assessment directs, drives, and influence learning. This understanding holds the key for an educator, teacher, or policymaker to direct the entire process of learning through appropriate assessment tools.[1]

One of the models of competency, the Miller's Pyramid, provides a basic framework for bringing into focus the four levels of competence: ‘Knows', ‘Knows how', ‘Shows how', and ‘Does’ (Miller, 1990).[2] Multiple choice questions or short answered questions or long answered questions can test the first two levels while the latter two components can be tested by objective structured practical/clinical examination (OSPE/OSCE) and other performance-based assessment tools.

In India, the Postgraduate Medical Education Regulations 2000 of the Medical Council of India (MCI) states that postgraduate training be competency-based and also suggests the use of ‘Logbook’ for monitoring the learning process. However, these regulations do not provide any process or road map of in-training assessments.[3] There are few gray areas in the document of Graduate Medical Regulations of MCI 1997 like ‘integrated teaching', which has got somewhat disappointing results due to lack of provision of operational details or/and monitoring details.[4] While there is a provision of internal assessment for periodic assessments during the undergraduate (UG) program in Indian Medical Regulations, there are no such pre-requisites for postgraduate (PG) courses in India.

PG training is directed not merely at the attainment of knowledge, attitude, and skills but also observable responsiveness and appropriate functioning in real-life situations. The purpose of PG education is to create specialists who would provide high- quality healthcare and advance the cause of science through research and training. At the same time, they would also serve as physicians of the first contact for the community, practicing affordable, accessible primary healthcare to the rural, needy, marginalized, and underprivileged populace.

It follows that even the most ideal of conventional assessments conducted in examination settings will fall short of measuring these outcomes. There is indeed a need to observe and assess the trainees in real situations so that necessary corrections can be provided to the trainees.[5] Observational assessment (OA) or Workplace-based assessment (WPBA) is being increasingly used to assess the trainees by direct observation and to shape their learning.[6],[7] The current deficiencies in the assessment system of India are due to a lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools. Hence, considering the Indian scenario, using the advantages of OA over the traditional assessment methods and its educational utility; the study aimed to introduce formative assessment (FA) in practicals for PGs at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (DU), Wardha (MS), India. The objectives of the study were to train and sensitize the faculty/supervisors and PG trainees for FA in PG practical as “Assessment for Learning” and to evaluate the program for its feasibility and effectiveness.


  Materials and Methods Top


The study was conducted from June 2015 to October 2016, in all the Departments (having Postgraduate courses) of Medical, Dental, Ayurveda, and Nursing Colleges, of DMIMS (DU), Wardha (MS), India. The study participants were the PG students of the first and second academic year and faculty/guides/supervisors of the respective departments. The approval from Institutional Ethics Committee was obtained (Ref No.: DMIMS (DU)/IEC/2015-16/155).

It was an interventional prospective analytical study. OA tools used for FA were mini clinical evaluation exercise (MiniCEX) and direct observation of procedural skills (DOPS) in addition to OSCE/OSPE. Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (preclinical, paraclinical, clinical). There was the flexibility of the sites for using these tools as outpatient department (OPD) setting, in-patient department (IPD) settings, community/fieldwork, operation theater etc., as per the requirement of the subject/department.

Sensitization of the faculty was done for the concept of ‘FA in practical for postgraduate'. The faculty was trained for using the tools for OA, i.e., MiniCEX and DOPS as well as in providing developmental feedback on direct observation. The training was given by conducting a one-day workshop for the faculty. Similarly, the PG students were also sensitized by the respective Heads of the Department regarding the importance and implementation strategy of FA in practical.

All the departments identified various competencies and skills to be attended by the junior residents and prepared checklists for OA as well as various stations for OSPE/OSCE. Procedural skills were identified as per the level of residency of the PG students. All the skills or competencies identified by subject experts to be used for MiniCEX or DOPS and OSCE/OSPE stations were prevalidated by the respective Heads of the Department followed by rigorous validation by faculties of the Department of Assessment and Evaluation, School for Health Professions Education and Research.

The scores assigned for each FA (per semester) in practical for PGs were as follows:

  • 3 OAs at the end of first semester = 30 marks
  • 3 OAs at the end of second semester = 30 marks
  • OSPE/OSCE examination (8 stations) =40 marks


  • (Total of 100 marks for each academic year).


Later, as per the operational strategy for FA in practical, validated tools for identified skills were implemented for FA in practical at the end of each semester. The result was prepared by the respective Department and submitted for final analysis. Feedback from PG teachers/supervisors and PG students were solicited for the feasibility, practicality, and applicability of FA in practical for PGs in the form of the validated structured feedback proforma. All the enrolled PGs (n = 331) and all registered PG teachers/supervisors (n = 153) participated in the study.


  Results Top


Analysis of marks/scores obtained in Formative assessment

[Table 1] depicts the mean scores of the FA (OA and OSPE/OSCE) of the PGs of various institutes under the ambit of the university. The mean scores of Junior Resident 1 (JR1) and Junior Resident 2 (JR2) are calculated for the FA. Overall,
Table 1: Institution wise distribution of marks of Formative assessment (Observational assessment and OSPE/OSCE) of the postgraduates

Click here to view


  • The mean scores of FA ranged in between 58.11 and 64.18 out of 100 marks
  • The highest score of 64.18 marks out of 100 is of the JR2 of medical college
  • For the institute of Ayurveda, the mean FA score for JR1 was 50.21/80 marks and for JR2 it was 75.83/120 Marks
  • In all the institutes, for both JR1 and JR2, mean scores of 1st OA were improved in the 2nd OA test
  • The mean scores of OSPE/OSCE were also above 50% for both JR1 and JR2 in all the institutes.


Line diagrams [Figure 1] depicting the trend of the mean scores of OA for JR1 of Medical college showed improvement in scores.
Figure 1: Trend of the scores of observational assessment for JR1 of Medical college

Click here to view


Out of all the departments of medical faculty, having PG students (JR1) of the Batch 2015-16, in 4 departments (ENT, Medicine, Pediatrics, Radiology) the mean FA score came below 50%. Similarly, for JR2 of the batch 2014-15, in one department (Community Medicine) the mean FA score was found to be below 50% [Figure 2] and [Figure 3].
Figure 2: Department-wise distribution of mean scores of formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR1) of Medical college

Click here to view
Figure 3: Department wise distribution of mean scores of Formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR2) of Medical college

Click here to view


On similar lines, out of all the departments of dental faculty having PG students (JR1) of the Batch 2015-16, in only one department (Oral Medicine & Radiology) the mean FA score came below 50%. For JR2, all the departments of dental faculty had the mean FA score above 50% [Figure 4].
Figure 4: Score of Formative Assessment for JR1 of Dentistry

Click here to view


Total FA score of all departments of Ayurveda faculty for JR1 is calculated considering one OA test and OSPE/OSCE marks (as the first OA was not conducted due to late admissions. See Table No. 1). The total marks are out of 80 for JR1 of Ayurveda. For JR1, all the departments of Ayurveda faculty had the mean formative assessment score above 50%. All the departments of Ayurveda College, having PG students (JR2) of the Batch 2014-15, the mean FA score was above 50% (>60 marks).

For the faculty of Nursing, in all the departments, the JR1 scored above 50% for the FA.

Feedback analysis for Formative assessment

The response rate of feedback from all PG students was 92% whereas from PG teachers/supervisors of all the constituent colleges was 79%. For almost all the parameters of feedback, the PG students showed satisfaction towards the various aspects of FA. The rating average ranged between 3.86 and 4.19 [Figure 5]. Overall rating by the PGs and faculty for the process of FA showed that cumulatively 88.27% PGs and faculty rated the FA as Good (72.22%) to Best (16.05%).
Figure 5: ‘Rating average’ for feedback from postgraduate students and faculty regarding FA (measured on 5 point Likert scale)

Click here to view


Key suggestions to improve Formative assessment in the department given by students and teachers

  1. FA provides a good opportunity for PGs to self-assess their practical knowledge
  2. Should be undertaken every fortnightly or monthly
  3. Helps to improve the PG's practical and communication skills with patients
  4. Should be included in summative assessment too
  5. OSCE should be done in both IPD and OPD settings
  6. Combination of OSCE, MiniCEX/DOPS should be applied to every patient's procedure (minor surgical) for the betterment of results
  7. More procedures to be covered through DOPS/MiniCEX
  8. It should not only be taken for exam purpose but also taken once in a week to improve patient examination skills
  9. Should be implemented as a routine day to day system (such as during rounds or immediate after wand rounds)
  10. Supervised learning may be useful
  11. For improvement in FA, only procedural skills should not be chosen for DOPS but it should include problem-solving skills in the form of case scenario and differential diagnosis, clinical application of the procedure, etc
  12. Repeated conduct should be monitored for improvement in outcome.


Hindrances faced by postgraduate students and faculty during the implementation of Formative assessment in the department

Majority of the PG students opined that the FA was beneficial and there were no significant hindrances during its implementation.

Some of the hindrances informed by others were as follows: -

  1. Time constraint, Space constraint, in some departments
  2. If it is conducted in routine clinical working hours, there is a rush by both PGs and faculty to get back to the routine clinical duties. Hence, FA should be taken in a separate time slot
  3. Need to have many more faculty/assessors available in the department on the day of FA, in major clinical departments with a significant number of PGs
  4. It is tedious to prepare a checklist for each and every procedure. If there is no trained staff for how to prepare the checklist or how to conduct DOPS, then responsibility lies with only trained staff in the department
  5. More time is required for doing all things meticulously. Because of the task of completing clinical and/or departmental work, faculty may not give much time for everyone during FA.



  Discussion Top


Health educators have differentiated assessment according to its purpose; assessment for learning (FA) and assessment of learning (Summative assessment). When the information from an assessment is used solely to make a judgment about the level of competence or achievement, it is the summative assessment. FA refers to assessment that is particularly intended to provide feedback on performance to improve and augment learning. It is not the instrument that is formative, it is the use of the information gathered, by whatever means, to adjust teaching and learning that merits the “formative” label.

Post Graduate Medical Education Regulation 2000 document[8] of the Medical Council of India (the regulatory body of Medical Education in India) mentions that the PG curriculum should be ‘competency-based'. It should ideally develop competent, confident, concerned, compassionate, and globally relevant healthcare professionals who must provide comprehensive healthcare. However, there is a need for specific guidelines regarding assessing those competencies. At present, the three- year long PG training is assessed at the endpoint only (summative assessment). This means that the assessment is done for the product and not for the process. There are very few colleges/Universities conducting FA in theory for PG in India. But very little literature is available for FA especially for assessing competency through practical, for all health professions.

DMIMS (DU), Wardha is a Deemed University (in Rural setting) having Medical, Dental, Ayurved, and Nursing Colleges, which are regulated by their respective Apical Councils. These councils do not encompass any form of formative assessment or continuous internal assessment in their PG training program. Knowing the importance of FA for learning, we conducted a study for PG students during the academic year 2015-16 for practical (in the domain of skills and attitude) and tested for its feasibility and effectivity.

In this study, it was observed that the mean score of OA conducted for the second time was improved as compared to the first OA test scores. Though these scores were not statistically significant the increasing trend was observed for both JR1 and JR2. OSPE/OSCE was one of the modalities of the FA. OSCE was used as a tool for both formative and summative assessment of Japanese PG in clinical training at Hiroshima,[9] wherein it was found that trainees viewed OSCEs positively and appreciated their effectiveness from a pedagogical perspective, and OSCE positively affected the trainees’ approach to learning. Similarly, Miller JK,[10] in his study, on ‘Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy', an adaptation of the OSCE procedures was done for competency-based training of Marriage & Family Therapy students. Instead of using the procedures as a summative examination as is typical in medical education, the study proposed how to use them as formative exercises in the development of student competence.

The study by Dijksterhuis M G et al.[11] aimed to explore perceptions of PG and teachers about factors that determine active engagement in FA. It was stated that engaging in FA with a genuine impact on learning is complex and quite a challenge to both students and teachers. Increased acceptability along with the effective implementation of FA structure, individual perspectives on feedback, a supportive learning environment, and credibility of feedback are all important in this process. Lases LSS et al. (2019)[12] too opined that a healthy learning environment is associated with improved resident well-being. Every one of these should be taken into account when the utility of FA in PG medical training is evaluated. On the other hand, as per Watling and Ginsburg (2019),[13] the emergence of learning from a cauldron of assessment and feedback can seem like alchemy. Whereas feedback stresses development and learning, assessment stresses judgment and decision making. Mixing these distinct ingredients creates a risky recipe for learning.

In this study, all the PG residents and teachers/assessors were asked to give feedback regarding the feasibility and utility of this program in a structured questionnaire. Feedback analysis depicted the acceptance of FA by majority of students and faculties.


  Conclusions Top


This study of ‘Formative Assessment in Practical for postgraduates in Health profession at DMIMS (DU) Wardha (MS), India', revealed that it was a feasible, acceptable, and effective program (with minor modifications at respective departments) to improve the overall learning of students. From the results, it can be concluded that the overall response from the PG students and faculty for FA was good. The results showed a significant improvement in scores from one semester to other as the PG learner progressed.

Limitations

As it was a pilot study conducted only for 1 year, the prospective progress of the PG student could not be computed.

Recommendations

Looking at the need for FA for Learning in Health Education, the following recommendations are made.

  1. To include OSCE/OSPE in summative assessment of PG training
  2. Increasing the number of OAs to have more reliability and thus increasing validity
  3. Incorporation of FA (practical) in PG training program as ELIGIBILITY CRITERIA (minimum 35% or 50%) for progression
  4. Inclusion of the concept of Continuous Internal Assessment in Post Graduate Assessment, as in the case of undergraduates
  5. It should be included in the curriculum as a part of the scheme of examination
  6. Bank of OSCE station for PG should be prepared at the departmental level.


Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.



 
  References Top

1.
Adkoli BV. Assessment of professionalism and Ethics. In: Singh T, Anshu, editor. 1st ed. Jaypee Brothers Medical Publishers: New Delhi; 2012. p. 180-190.  Back to cited text no. 1
    
2.
Miller G. The assessment to clinical skills, competence, performance. Acad Med 65;1990:S63-7.  Back to cited text no. 2
    
3.
Medical Council of India. Postgraduate Medical Education Regulations, 2000. Available from: http://www.mciindia.org. [Last accessed on 2015 Mar 11].  Back to cited text no. 3
    
4.
Medical Council of India. Graduate Medical Regulations, 1997. Available from: http://www.mciindia.org. [Last accessed on 2015 Mar 11].  Back to cited text no. 4
    
5.
Norcini JJ, Burch V. Workplace based assessment as an educational tool: AMEE guide No. 31. Med Teach 2007;29:855-71.  Back to cited text no. 5
    
6.
Downing SM, Yudkowsky R. Observational Assessment; Goals and Tools. Assessment in Health Professions Education Routledge Publication, New York. First Ed 2009.  Back to cited text no. 6
    
7.
Van der Vleuten CPM, Schuwrith CWT. Assessing professional competence from methods to program. Med Educ 2005;39:309-17.  Back to cited text no. 7
    
8.
Medical Council of India Postgraduate Medical Education Regulations, 2000, (amended upto January 2017), Medical Council of India Pocket – 14, Sector 8, Dwarka, NEW DELHI – 110 077. Available from: http://www.mciindia.org.  Back to cited text no. 8
    
9.
Taguchi N, Ogawa T. OSCEs in Japanese postgraduate clinical training Hiroshima experience 2000-2009. Eur J Dent Educ 2010;14:203-9.  Back to cited text no. 9
    
10.
Miller JK. Competency-based training: Objective structured clinical exercises (OSCE) in marriage and family therapy. J Marital Fam Ther 2010;36:320-32.  Back to cited text no. 10
    
11.
Dijksterhuis MG, Schuwirth LW, Braat DD, Teunissen PW, Scheele F. A qualitative study on trainees’ and supervisors’ perceptions of assessment for learning in postgraduate medical education. Med Teach 2013;35:e1396-402.  Back to cited text no. 11
    
12.
Lases LSS, Arah OA, Busch ORC, Heineman MJ, Lombarts K. Learning climate positively influences residents’ work-related well-being. Adv Health Sci Educ Theory Pract 2019;24:317-30.  Back to cited text no. 12
    
13.
Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ 2019;53:76-85.  Back to cited text no. 13
    


    Figures

  [Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5]
 
 
    Tables

  [Table 1]



 

Top
   
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
   Abstract
  Background
   Materials and Me...
  Results
  Discussion
  Conclusions
   References
   Article Figures
   Article Tables

 Article Access Statistics
    Viewed25    
    Printed0    
    Emailed0    
    PDF Downloaded17    
    Comments [Add]    

Recommend this journal