World Rural Health Conference
Home Print this page Email this page Small font size Default font size Increase font size
Users Online: 97
Home About us Editorial board Search Ahead of print Current issue Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents 
ORIGINAL ARTICLE
Year : 2018  |  Volume : 7  |  Issue : 6  |  Page : 1458-1463  

Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study


1 Department of Family Medicine, Community Health and Development, Christian Medical College, Vellore, Tamil Nadu, India
2 Department of Family Medicine, Low Cost Effective Care Unit, Christian Medical College, Vellore, Tamil Nadu, India
3 Department of Family Medicine, Shalom Family Medicine Clinic, Christian Medical College, Vellore, Tamil Nadu, India

Date of Web Publication30-Nov-2018

Correspondence Address:
Yeshvanth Kumar
Department of Family Medicine, Low Cost Effective Care Unit, Schell Eye Campus, Arni Road, Christian Medical College, Vellore - 632 001, Tamil Nadu
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jfmpc.jfmpc_141_18

Rights and Permissions
  Abstract 


Background: Department of Family Medicine in a medical college in South India introduced “field note” (FN) as a tool for Work-Place Based Assessment in postgraduate training. FN collects “open-ended” feedback from both resident and faculty and helps them to arrive at an action plan. This study describes our experience of implementing FN and perceptions of learners and faculty. Methods: While precepting the residents in Family Medicine service areas, faculty documented their observations of the resident's clinical work using FN and provided an action plan. Faculty and residents described their experience and provided feedback. Focus group discussions were conducted for faculty and residents. Data were coded and grouped into themes. Results: Four residents and seven faculties participated in the study during 12 weeks period using 17 consultations. Clinical expert (13/17) and communicator (6/17) are the most commonly assessed competencies followed by professionalism (2/17) and collaborator (2/17). Faculty and residents agreed that “FN” was a useful tool and it helped the faculty to give feedback and guide the learner. Residents and faculty arrived at an action plan in 70% of the consultations. Three of four residents perceived the change in their behaviour positively after the use of FN. Both resident and faculty found the rating of the learner using Dreyfus scale as a barrier. Conclusion: FN could be one of the important tools in our “Toolbox of Assessment Methods” for family medicine specialty. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.

Keywords: Competencies, family medicine, field note tool, formative assessment, postgraduate education


How to cite this article:
Mathew AE, Kumar Y, Angeline RP, Christopher P, Rehman SP, Venkatesan S. Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study. J Family Med Prim Care 2018;7:1458-63

How to cite this URL:
Mathew AE, Kumar Y, Angeline RP, Christopher P, Rehman SP, Venkatesan S. Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study. J Family Med Prim Care [serial online] 2018 [cited 2018 Dec 18];7:1458-63. Available from: http://www.jfmpc.com/text.asp?2018/7/6/1458/246454




  Introduction Top


“Assessment drives student learning” is a known axiom in medical education. Even though assessments use more resources, it improves motivation of the learner and helps the faculty and learner to achieve educational outcomes.[1],[2] While summative assessment is effective in motivating students to improve their performance toward certification, it cannot be used to provide feedback and guide the learner when a deficiency is identified.[3] Work-Place Based Assessment (WPBA) assesses the “does” level of competence in Miller's pyramid. It is done in an authentic environment and in the context where the resident is going to work in future. It is not merely intended to assign a grade to the learner but to provide feedback and guide the learner. This should be an ongoing activity, based on direct observation by the faculty. It clarifies the goals with the learner and helps faculty to diagnose and correct specific mistakes in learning and behavior.[4]

Competency-based education is being introduced in family medicine postgraduate training across the world.[5] Workplace-based assessments are considered more authentic for assessment of competence and performance in real life. There are many tools available for WPBA like Mini-clinical evaluation exercise, video-review, case-based discussion, multi-source feedback, and portfolio.[6],[7] Field note (FN) is a new tool that has been described by Ross et al. They used FN as the primary assessment tool in the Competency-Based Achievement System in the University of Alberta, to provide feedback to the residents.[8]

FN is a paper or electronic document in which both resident and faculty reflect on the consultations done by the resident [Table 1]. Open-ended feedback is collected about the competencies demonstrated by the resident in the clinical encounter. FN helps the resident to arrive at an action plan in discussion with the faculty. In FN, the learner needs to reflect first followed by the faculty and this is documented and stored for later review.
Table 1: Reflective questions in the Field note tool; (created based on the Competency-Based Achievement System, used by University of Alberta in Edmonton[8]

Click here to view


This study is aimed at introducing “field note” as a tool for workplace-based assessment in family medicine and collecting the perceptions of the learners and faculty.


  Method Top


After obtaining ethics committee approval and informed consent, four residents and seven faculties were sensitized to the FN tool and the definitions of competencies in family medicine. Over 12-week study period (March to June 2016), the faculty observed and documented the clinical consultations by the resident in the FN and rated the competencies using “Dreyfus five stage model of adult skill acquisition.”[9] The seven competencies that were assessed are the clinical expert, communicator, professional, collaborator, system-based practitioner, scholar, and leader. These competencies were defined by the faculty attending the National Faculty Development Workshop organized by National Board of Examinations in Christian Medical College, Vellore in February 2016 and were adopted from ACGME Family Medicine Milestone project [Table 2]. The ratings used were (ascending from lower to higher level of performance) novice, advanced beginner, competent, proficient, and expert where novice represents the learner without any knowledge about the competency and expert represents the learner who has demonstrated the maximum level of performance in that competency. In our study, we used a paper-based two-page FN document that was simple to use and could be filed for future reference.
Table 2: Definitions of competencies assessed on the residents*

Click here to view


Faculty provided feedback about the selected few competencies relevant for each consultation and helped the learner to arrive at an action plan to progress in their learning curve. Faculty and learners were asked to describe their experience of using the FN and rate the usefulness of FN for assessing the learner and for giving appropriate feedback using the Likert scale.

Both faculty and residents suggested changes for improvement and mentioned about action plan they provided or received. Focus group discussions were conducted for faculty and residents to collect their perceptions and challenges in using the FN. The data were coded and grouped into themes. The following questions were asked to focus group: describe your feelings and experiences on using FN, any challenges or barriers you faced while using FN, and any suggestions to make it a better tool.


  Results Top


Four learners and seven faculties participated in the assessment using FN during 12 weeks study period, using 17 consultations. “Clinical expert” was the commonly assessed competency (13 of 17 consultations) followed by “communicator” (6/17), “collaborator” (4/17), “professional” (2/17), “scholar” (2/17), and “system-based practitioner (2/17).” “Leader”as a competency was never assessed during the study period. In each consultation, the faculty assessed one or more competency, decided by the faculty.

“Advanced beginner”was the most common rating given by the faculty in 7 of 17 consultations, followed by competent (5/17) and novice (2/17). Residents were never rated as “proficient” or “expert” in any of the consultations. In 3 of 17 encounters, there was no rating mentioned at all. In about 70% of (12 of 17) consultations, residents were given action plan. Three of four residents perceived a positive change in their behavior in achieving the family medicine competencies.

All faculty and learners agreed that the FN is a useful tool for providing feedback and guiding the learner. Overall, FN was found to be a useful tool for assessment of the learner's competence. Almost half of the residents were “hesitant” to agree on the usefulness of the FN [Figure 1], [Figure 2], [Figure 3]. [Table 3] and [Table 4] describe the selected quotes from the faculty and learners, respectively. The quotes are grouped into strengths and challenges/barriers.
Figure 1: Usefulness of “field note” to provide feedback

Click here to view
Figure 2: Usefulness of “field note” to guide learners

Click here to view
Figure 3: Usefulness of the tool for formative assessment

Click here to view
Table 3: Quotes from the faculty focus group

Click here to view
Table 4: Quotes from the residents focus group

Click here to view


Suggestions from the faculty and resident

The following are few suggestions from the faculty;

  • “There is a need for entry level assessment of the competencies in the PG student”
  • “Systems are needed for behavioral change in resident when a problem is identified”
  • “Action plan needs to be followed up”
  • “If we could make the documentation online and review periodically, it would be better.”


The following are the suggestions from the resident:

  • “Give time to reflect with the resident”
  • “No need of rating of the student”
  • “Field note should be done for the same condition/problem over few weeks if possible to assess whether the learner has improved based on the feedback by the faculty”
  • “A learner needs to be followed up. If a learner is Novice in one topic, he could be followed up with multiple field notes till he is competent on that topic”
  • “Remove the Novice and advanced beginner. Rather use stage 1, 2, 3, and 4”
  • “Is it possible to counter check the action plan? Whether the action plan is achieved?”
  • “One faculty could see all the field notes according to the subject and according to the competencies and provide overall feedback to the resident.”



  Discussion Top


Any assessment method should be aligned with the content of the training program and all the desired competencies of the specialty. Assessment is also expected to have an educational impact.[10] In postgraduate education in India, more emphasis is given for summative assessment using theory examination and practical clinical case discussions. In these high-stakes summative examinations, only a few competencies such as clinical expertise and communication are assessed. But there is less opportunity to assess important competencies like the collaborator, leader, scholar, and system-based practitioner which are needed for the practice of medicine.

This pilot study found that “FN” can be used to assess most of the competencies in family medicine. Leader as a competency was never assessed in our study. That could be explained by the short duration of the study. Leader is a complex competency and it needs multiple observations to make a reliable judgement. Formative assessment using FN is a way forward in implementing competency-based medical education.

FN was accepted as a tool for workplace-based assessment both by learners and faculty, and it is cost-effective. But the learners had difficulty in accepting the negative feedback and the rating as novice or advanced beginner. They had received more negative feedback than positive. This could be one of the reasons for their hesitation in agreeing on the usefulness of FN [Figure 1], [Figure 2], [Figure 3]. The residents have felt the differences in the ways they are assessed by different faculties [Table 4]. There is a need for developing definitions of milestones of competencies and faculty development in assessment and feedback to reduce the observer bias.

Educational impact of the FN was perceived by the residents during the focus group discussion [Table 3] and [Table 4]. The utility of any assessment tool depends on its validity, reliability, educational impact, acceptability, and cost. We did not measure the “validity” of the tool in this study. Validity refers to whether the instrument measures what it is expected to measure. According to Dr. Van der Vleuten, validity depends on the authenticity of measurement and integration of competencies during measurement.[11] FN was used to assess all competencies, in any clinical context: being outpatient, in-patient care, emergency care, and home care. FN can be used to assess the consultation skills, procedural skills, ward rounds, counselling skills, and academic presentations. The fact that FN is used in the authentic workplace and the whole task is assessed without trivialization of the competencies, FN can fulfil the criteria for validity.

“Reliability” is the reproducibility of the scores achieved in assessment.[11] One can argue that assessment by FN has the subjective bias as it is open-ended without objective scores and this may negatively affect reliability. Van der Vleuten says the reliability of an instrument depends on careful sampling across the content and not based on objectivity or standardization. To reduce the observer bias and improve the reliability, we decided to collect the FN of each resident given by different faculties at different context and file it for review by the course coordinator once in 4 months to assess the resident's progress. A feedback will be given to the resident about the performance in the previous 4 months and also suggestions will be given to achieve the desired milestones in the next 4 months.[8]

Limitations

This is a pilot study done over a short period in one institution in India. Findings of this study need to be validated in a larger multicentric study done in multiple settings with many institutions training family medicine residents in India.


  Conclusion Top


FN could be one of the important tools in our “Toolbox of Assessment Methods” for family medicine specialty when we introduce competency-based postgraduate medical education in India. There are few challenges observed in using the FN. Rating the learner using Dreyfus levels was the most important barrier that needs to be addressed during implementation. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.[13]

Acknowledgement

The authors would like to thank all the four residents who were pursuing Fellowship in Family Medicine in Christian Medical College, Vellore, and who participated in the study.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.



 
  References Top

1.
Ferris H, Flynn DO. Assessment in medical education; what are we trying to achieve? Int J Higher Educ 2015;4.  Back to cited text no. 1
    
2.
Bennett, RE. Formative assessment: A critical review. Assessment in Education Principles, Policy & Practice 2011;18:5-25.  Back to cited text no. 2
    
3.
Raupach T, Brown J, Anders S, Hasenfuss G, Harendza S. Summative assessments are more powerful drivers of student learning than resource intensive teaching formats. BMC Med 2013;11:61.  Back to cited text no. 3
    
4.
Thomas JW, Megan SK, Lutana HH. Formative assessment in health care education. Int J Educ Soc Sci 2014;1.  Back to cited text no. 4
    
5.
Tannenbaum D, Kerr J, Konkin J, Organek A, Parsons E, Saucier D, et al. Triple C Competency-Based Curriculum. Report of the Working Group on Postgraduate Curriculum Review – Part 1. Mississauga ON: College of Family Physicians of Canada; 2011.  Back to cited text no. 5
    
6.
Epstein RM. Assessment in medical education. N Engl J Med 2007;356:387-96.  Back to cited text no. 6
    
7.
Yousuf Guraya S. Workplace-based assessment; applications and educational impact; Malays J Med Sci 2015;22:5-10.  Back to cited text no. 7
    
8.
Ross S, Poth CN, Donoff M, Humphries P, Steiner I, Schipper S, et al. Competency-based achievement system: Using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician 2011;57:e323-30.  Back to cited text no. 8
    
9.
Dreyfus SE. The five-stage model of adult skill acquisition. Bull Sci Technol Soc 2004;24:177.  Back to cited text no. 9
    
10.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29: 855-71.  Back to cited text no. 10
    
11.
van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programs. Med Educ 2005;39:309-17.  Back to cited text no. 11
    
12.
Accreditation Council of Graduate Medical Education. Available from: https://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/NextAccreditationSystem/Milestones.aspx. [Last accessed on 2017 Jan].  Back to cited text no. 12
    
13.
Allen S, Anim T, Anthony E, Araujo D, Beebe D, Dostal J, et al.; The Family Medicine Milestone Project. A Joint Initiative of “The Accreditation Council for Graduate Medical Education” and “the American Board of Family Medicine”; October 2015. Available from: https://www.acgme.org/Portals/0/PDFs/Milestones/FamilyMedicineMilestones.pdf. [Last accessed on 2017 Jan].  Back to cited text no. 13
    


    Figures

  [Figure 1], [Figure 2], [Figure 3]
 
 
    Tables

  [Table 1], [Table 2], [Table 3], [Table 4]



 

Top
   
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
   Abstract
  Introduction
  Method
  Results
  Discussion
  Conclusion
   References
   Article Figures
   Article Tables

 Article Access Statistics
    Viewed57    
    Printed1    
    Emailed0    
    PDF Downloaded17    
    Comments [Add]    

Recommend this journal