World Rural Health Conference
Home Print this page Email this page Small font size Default font size Increase font size
Users Online: 4997
Home About us Editorial board Search Ahead of print Current issue Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents 
ORIGINAL ARTICLE
Year : 2015  |  Volume : 4  |  Issue : 4  |  Page : 539-545  

Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges


1 Department of Community Medicine, School of Public Health, PGIMER, Chandigarh; International Union Against Tuberculosis and Lung Disease, The Union South East Asia Office, New Delhi, India
2 Department of Community Medicine, School of Public Health, PGIMER, Chandigarh, India
3 Department of Community Medicine and Family Medicine, AIIMS, Bhubaneswar, Odisha, India
4 Public Health Consultant, National Health Mission, Chandigarh, India

Date of Web Publication18-Jan-2016

Correspondence Address:
Jaya Prasad Tripathy
Department of Community Medicine, School of Public Health, PGIMER, Chandigarh
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/2249-4863.174282

Rights and Permissions
  Abstract 

Background: Community monitoring was introduced on a pilot mode in 36 selected districts of India in a phased manner. In Chandigarh, it was introduced in the year 2009-2010. A preliminary evaluation of the program was undertaken with special emphasis on the inputs and the processes. Methodology: Quantitative methods included verification against checklists and record reviews. Nonparticipant observation was used to evaluate the conduct of trainings, interviews, and group discussions. Health system had trained health system functionaries (nursing students and Village Health Sanitation Committee [VHSC] members) to generate village-based scorecards for assessing community needs. Community needs were assessed independently for two villages under the study area to validate the scores generated by the health system. Results: VHSCs were formed in all 22 villages but without a chairperson or convener. The involvement of VHSC members in the community monitoring process was minimal. The conduct of group discussions was below par due to poor moderation and unequal responses from the group. The community monitoring committees at the state level had limited representation from the non-health sector, lower committees, and the nongovernmental organizations/civil societies. Agreement between the report cards generated by the investigator and the health system in the selected villages was found to be to be fair (0.369) whereas weighted kappa (0.504) was moderate. Conclusion: In spite of all these limitations and challenges, the government has taken a valiant step by trying to involve the community in the monitoring of health services. The dynamic nature of the community warrants incorporation of an evaluation framework into the planning of such programs.

Keywords: Chandigarh, community monitoring, process evaluation


How to cite this article:
Tripathy JP, Aggarwal AK, Patro BK, Verma H. Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges . J Family Med Prim Care 2015;4:539-45

How to cite this URL:
Tripathy JP, Aggarwal AK, Patro BK, Verma H. Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges . J Family Med Prim Care [serial online] 2015 [cited 2019 Nov 20];4:539-45. Available from: http://www.jfmpc.com/text.asp?2015/4/4/539/174282


  Introduction Top


The Alma Ata Declaration [1] (1978), Ottawa Charter [2] (1986), and Agenda 21 [3] (1992) among others have placed community participation high on the political and public health agendas of member nations. National Rural Health Mission (NRHM), Chandigarh, India also spells clearly the importance of community participation as part of the decentralized process of health care management. [4]

Community Monitoring under NRHM was implemented in a phased manner in India. In Chandigarh, it was introduced in the year 2009-2010. It empowers the community to work with the health service providers to identify and solve the community problems. Government of India (GOI) has issued comprehensive guidelines for the same. [4] States were supposed to adopt the guidelines keeping the spirit of community representation intact.

All implementing states should review this novel but challenging strategy periodically. However, the methodology, tools, and experiences for its evaluation are very limited. Thus, we undertook a process evaluation of the community monitoring implementation in Chandigarh with the following objectives: (a) How well organizational structures for community monitoring were set up (b) How were trainings organized and how much knowledge and skill improvements occurred with the trainings (c) to what extent communities participated in the problem assessments and had discussion with the health service providers (d) to understand the gaps in implementation, and (e) to ascertain the feasibility of the methods and tools used to measure implementation of community monitoring.


  Methodology Top


The study was carried out in the union territory of Chandigarh. Mixed methodology approach was followed. Both quantitative and qualitative methods were used for evaluation. Methods such as desk review, in-depth interviews, checklists, nonparticipant observation, training evaluation, etc., were employed to assess various components of community monitoring implementation. The major structural and functional domains of the community monitoring process were evaluated, that is, organizational set up, quality of trainings, conduct of interviews and group discussions, and the involvement of the community in the entire exercise.

Organizational set up

A structured scoring checklist was developed to evaluate the monitoring committees at two levels, that is, state/union territory and village level namely the State Monitoring and Planning Committee, State Mentoring Committee and the Village Health and Sanitation Committee (VHSC). In-depth, interviews were conducted with the concerned nodal officer and records were reviewed to understand the constitution of the committees.

Quality of trainings

Records were reviewed to evaluate the quality of trainings viz., number and profile of participants, training methodology, and feedback. A checklist was developed for scrutiny of the training records. As very minimal record was available, it was not possible to comment on the quality of the training. However, during the course of the study, another training was scheduled which was evaluated by one of the investigators (JPT) who attended the training as a nonparticipant observer. Observations were recorded using a scoring checklist.

To assess the impact of the training, knowledge of the health system staff (nursing students and VHSC members) was assessed using a structured questionnaire. It had questions on entitlements under NRHM and preparation of report cards. A comparison was also made between the levels of knowledge between the trained (93) and the untrained (43) health system functionaries.

Conduct of the interviews and group discussions

The process of data collection viz., interviews with maternal beneficiaries and group discussions were evaluated by observation in two villages using a scoring checklist. The investigator (JPT) undertook nonparticipant observation using a checklist to observe the interviews and focus group discussions (FGDs) in these villages.

Validation of report card scores

Investigator (JPT) made independent visit to some houses and filled the scorecards himself to validate the findings with those filled by the health system staff who were trained for the same by the union territory health department. Investigator visited the houses about 15 days after the visit of the trained health staff. These scorecards are color coded with red, yellow, and green color codes signifying the urgency of the action, with red being most urgent and green as normal.

Kappa agreement between the color codes of the report cards generated by the investigator and the trained health staff in the same villages was also calculated.

Flow of information

This was assessed by review of records, scrutiny of village health report cards, and discussions with the program managers.

Ethical approval

The study was ethically approved by the Institute Ethics Committee, PostGraduate Institute of Medical Education and Research, Chandigarh, India.


  Results Top


Status of setting up of committees at state and village level as per Government of India guidelines

For setting up of state/union territory level committees as per the GOI guidelines, union territory Chandigarh achieved a score of 7 marks out of 20 (35%). State Monitoring and Planning Committee had no or minimal representation from the elected bodies, lower committees, civil society, or other departments [Table 1]. The mentoring committee at union territory level comprised of only three members, all of them being faculty members in the Department of Community Medicine in two medical colleges in the state. Thus, there was no representation from the state health department, nodal nongovernmental organization (NGO) as well from the civil society in the mentoring committee.
Table 1: Score sheet for organizational set up in Chandigarh under community monitoring


Click here to view


VHSCs are to be constituted at the village level for village level planning, which is a crucial step toward accomplishing the goal of community monitoring. Chandigarh has 22 villages under its jurisdiction. All the 22 villages in Chandigarh have a VHSC. Each VHSC in a village was assessed based on eight parameters. Of a total score of 176, the villages scored 100 (57%) [Table 2]. None of the VHSCs has a nominated chairperson and a convener. The majority (144, 84.2%) of the members are females. Auxiliary nurse midwife (ANM) is present in all VHSCs, and Anganwadi worker (AWW) is present in 21 (95%) of 22 villages. VHSCs in villages of Chandigarh have doctors and social workers in 18 villages (82%). Panch is a member of VHSC in 17 villages (77%). Half of the villages have SC/ST candidate in the committee whereas only 9 villages (41%) have a teacher in their committees.
Table 2: Score sheet for evaluating organizational set up at the village level


Click here to view


Quality of trainings and assessment of knowledge assessment of the trained health system functionaries

Training record reviews did not yield desired information about the quality of sessions, teaching methods used, training curriculum, and the training coordinator. However, participant observation of a training program revealed that training was done in good ambience. However, trainings were lecture-oriented with all the participants attending the training together giving a high trainer: Trainee ratio of 1:70. No skills were imparted for conduct of interviews and organization of FGDs. The training session on tools of monitoring and data collection were taken hurriedly with little or no discussion.

A total of 93 health system functionaries (nursing students), who had received the training, were administered a knowledge assessment questionnaire. Seventy-two percent (67/93) secured at least 50% marks on questions related to health entitlements under NRHM whereas only 51 (55%) students could score at least 50% marks when enquired about the Preparation of Report Cards. Sixty-two percent (58/93) students scored at least 50% marks overall . Questionnaire was also administered to 69 VHSC members. None of them scored at least 50% marks in overall knowledge and the preparation of report card section. However, 61% (42/69) VHSC members interviewed scored more than 50% marks regarding Health Entitlements under NRHM [Table 3].
Table 3: Comparison of knowledge scores among different categories of health system functionaries


Click here to view


Forty-three untrained staff (nursing students) of the same batch were also administered the same questionnaire. There was significant difference between the mean scores of trained staff compared to the untrained ones regarding preparation of report cards and the overall knowledge gained; for the questions on health entitlements under NRHM, no significant difference was seen in the mean scores of the two groups [Table 3].

The group discussions achieved a score of 50% (5/10) whereas the interviews scored 60% (6/10). The conduct of the group discussions was dismal with poor moderation and unequal responses from the group. The group discussions were more of a lecturing rather than being interactive.

The percentage of scores obtained by both categories of health system staff namely nursing students and VHSC members regarding preparation of report cards were 50% and 9.3%, respectively (P < 0.001). The difference in the scores about knowledge regarding health entitlements under NRHM was found to be insignificant. However, the overall mean score was found to significantly different between the two groups (P < 0.001) [Table 3]. Among VHSC members, the health-related members namely the ANM, AWW, and the teachers among the nonhealth members fared better than the other members.

Flow of information

The management information system in Chandigarh witnessed a deviation from the standard guidelines. The information after being collected at the village level goes directly to the state bypassing the VHSC. Thus, we have two models of MIS now (in the context of Chandigarh) as given in the figure [Figure 1].
Figure 1: Comparison of management information system models

Click here to view


A team of three consultants that is, two Public Health consultants from NRHM and faculty from the Department of Community Medicine, GMCH-32, Chandigarh visited all the villages and held meetings with the members of VHSC in the respective villages. They discussed major issues such as biomedical waste management, absence of doctor in the dispensary, utilization of VHSC and untied funds, JSY irregularities, poor health infrastructure, etc., Concrete remedial steps were taken to resolve some of these issues like opening of VHSC account, training to VHSC members regarding utilization of VHSC and untied funds, appointment of doctors. However, the administration was found wanting in issues like biomedical waste management and Poor infrastructure. Resolution of such issues needs a strong inter-sectoral coordinating framework which is grossly lacking in the organizational framework under Community Monitoring in Chandigarh.

Validation of the report cards

Investigator (JPT) independently visited some houses and filled the report cards. Kappa agreement between the color codes in the report cards generated by the investigator and the trained health staff was found to be to be fair (0.369) whereas weighted kappa (0.504) was moderate [Table 4].
Table 4: 3×3 table showing the agreement between the health system functionaries and the investigator


Click here to view



  Discussion Top


Evaluation of community-based activities is necessary for continuous improvement and sustainability of the program. [5] Community monitoring is an important strategy under NRHM. However, there are not much experiences available in the indexed literature about the methods and tools to monitor the implementation. In our study, we developed methods and tools and applied the same to evaluate the process of community monitoring. There were some important findings that could be useful to the program implementers and future researchers.

Involvement of community representatives and NGOs in committees at various levels was grossly missing which defeats the prime objective of the program. This may be due to the common notion among health officials that direct involvement of community can put the entire system under stress.

Organizational structure

The organizational set up in Chandigarh is a two-tier level as against the five-tier hierarchical structure proposed in the national guidelines. However, the unique administrative structure in Chandigarh well justifies the two-tier system. The State Level Monitoring and Planning Committee has little or no representation from the elected bodies, committees at lower levels, (i.e. VHSCs) nonhealth departments and the nodal NGO/civil society involved in community monitoring. Poor coordinating mechanism with other sectors and noninvolvement of NGOs and civil societies hinders effective decision making and implementing ability.

Maharashtra has been a successful model so far with adequate representation from the lower committee, NGO, health department, nonhealth sectors, and the elected ones in the committees at all levels. [6] The three-membered mentoring committee in Chandigarh, all of them being faculty members of medical colleges in Chandigarh, is in sharp contrast with that of Maharashtra which has 15 members, 11 eminent members from civil society organizations experienced in community-based work.

Among the different innovations tried, the recruitment of nursing students as the facilitating NGO was certainly one of them. The nursing students come from a health-related background and also there is little variation in the educational status, age and other personal attributes. Thus, it might be more effective to train such students rather than training people from diverse backgrounds mostly nonhealth.

Quality of trainings

Training quality assessment through participant observation proved very useful to assess the strengths and the gaps. Trainings were mostly lecture-oriented with no focus on skill building for interviews and FGDs. This got reflected in the poor conduct of interviews and FGDs in the community. The training should focus on community engagement, tools of community monitoring, preparation of report cards, conducting village health meetings, and group discussions. Field-based practical hands-on training and role plays might be useful.

Quality of group discussions

The quality of the group discussions observed was quite poor. They had females in the majority, and they were mostly the patients coming to the health facility or the AWW. This might confound the responses. The group discussions were more of a one-to-one lecturing rather than being an interactive discussion. Active participation from all the participants in a group discussion could not be elicited due to poor moderation. This necessitates field training or role play as part of training curriculum regarding conduct of interviews and group discussions.

Composition and functioning of village health sanitation committees

VHSC meetings were not being held regularly, and attendance of the meetings was also found to be poor. This might be due to the fact that the VHSCs in Chandigarh do not have a designated Chairperson and a convener. The uncertainty in the roles of the members of the committees was probably one of the reasons for the infrequent meetings of VHSCs. Doctors were observed to be the members of the village committees in 18 of 22 villages. Such precedence in VHSCs is not found in other parts of the country.

The NRHM evaluation study by Planning Commission in seven states also reiterates the fact that barring Tamil Nadu VHSCs are not discharging their duties as one would have liked to. [7] Another study in North India found poor awareness among the VHSC members about functions, members and activities of the committee and areas of utilization of the VHSC fund. [8] The Chandigarh health department is striving hard to strengthen the role of VHSCs in village health planning and action. The VHSC members have been trained recently about the preparation of village health plan, utilization of untied and VHSC funds, organizing regular meetings, and preparing meeting reports.

The poor knowledge of the VHSC members especially the nonhealth members regarding their roles and responsibilities and the preparation of report cards warrants a capacity building exercise to empower them to ensure their participation and guarantee quality health care services.

Involvement of the community

Engaging the community in planning and monitoring of health service delivery is central to enhancing the availability, accessibility, quality, and use of the public health system. [9] However, community monitoring in Chandigarh is working without its soul, that is, the community. The involvement of the community in this entire process is poor. Maharashtra through a couple of novel innovations tried to involve the community in the preparation of report cards. They converted the report card into a poster and put up in several places in the village so that everybody could participate in report card preparation process. Another innovation was the Arogya Jagruti Diwas, a long-day program where people used to gather in at a place and participate in the report card preparation process. [6]

Village Health Meetings and Jan Sunwais are forums where the report cards are shared with the community, and people share their experiences of seeking health care. These meetings actually ensure involvement of the user group or the general community. A diverse range of issues have been raised during Jan Sunwais, including: Availability of medicines, ambulance services, irregularities in the provision of incentives, corruption and illegal charging, attitude of the service providers, instances of denial of health services, etc., in the states of Maharashtra, Karnataka, and Rajasthan as part of community monitoring. But such community intensive activities could not be organized in any of the 22 villages in union territory, Chandigarh. Thus, the collated report cards, instances of denial of health care, and other local issues and grievances were never discussed in a public forum thereby failing to involve the community in the process.

Kappa agreement

Weighted kappa revealed moderate agreement between the report card scores compiled by the trained health staff and the investigator himself. The domains which showed good agreement were maternal health, child health, equipment, infrastructure, etc., while JSY payment, quality of care, unofficial charges, and disease surveillance yielded poor agreement (red when interviewed by the trained staff and yellow by the investigator). Probably the respondents failed to express themselves in front of the investigator but they reported grievances before the trained health staff. There can be some plausible reasons for this disagreement. The 15-day time gap between the data collection by the investigator and the trained health staff might have witnessed a withdrawal of a certain health care service which might have bred discontent among the respondents but that looks very unlikely. The maternal beneficiaries interviewed and the composition of the group discussions were different in both the cases which might have elicited different responses although we can argue that the community from which all the respondents have come is the same and have the same set of issues. Gender difference might also play a determining role in eliciting true responses. A group of female moderators/interviewers is more likely to generate true responses than a single male investigator particularly when the respondent is a female which is usually the case here. Thus, a team of two females moderating a group discussion at an Anganwadi center looks a better proposition. These domains are such that they require good rapport building with the community to extract true responses. This disagreement can be tackled if there are user group representatives actively involved in the process of data collection.

The tools and methods used in this study were similar to those suggested by Butterfoss who also mentioned methods such as participant surveys, key informant interviews, FGDs, observation of meetings, and review of documents to evaluate community-based studies. [10] Similar methodology was also used by another.

Limitations

Studies of community participation show promising short-term results but have mixed record of long-term success. This evaluation was done just 2 years after the implementation of the program. Thus, it being the initial phase might elicit the better response from the community. Moreover, structural, economic, cultural, and social constraints such as poverty, caste, class and gender hierarchies, lack of knowledge, access to health care, and weak health system hinder community participation. [11],[12],[13],[14] These factors could not be assessed in the present study.


  Conclusion and Recommendations Top


Early and long-term participation by community members is needed to sustain community-based projects. The dynamic nature of such programs warrants the incorporation of an evaluation framework into the planning of such programs. [5] Community-based monitoring in Chandigarh failed up to its expectations when it came down to effective engagement of the community. Monitoring and Planning Committees should include representation from NGOs/civil society and lower committees such as VHSCs. Nonhealth VHSC members and other key persons of the village to be involved in the process of data collection to promote greater involvement of the community and other stakeholders. Operations research needs to be carried out to make the tools simpler, understandable and contextual. Practical hands-on field training and role plays should be the part of the training curriculum with more focus on community engagement, organizing and conducting group discussions, and Jan Sunwais.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

 
  References Top

1.
WHO. Declaration of Alma-Ata. Geneva: World Health Organization; 1978. Available from: http://www.euro.who.int/__data/assets/pdf_file/0009/113877/E93944.pdf. [Last accessed on 2015 Jul 06].  Back to cited text no. 1
    
2.
WHO. Ottawa Charter for Health Promotion. Geneva: World Health Organization; 1986.  Back to cited text no. 2
    
3.
United Nations Sustainable Development. Proceedings of United Nations Conference on Environment and Development-Agenda 21, Brazil, Rio De Janeiro. United Nations; 1992. Available from: http://www.sustainabledevelopment.un.org/content/documents/Agenda21.pdf. [Last accessed on 2015 Jul 06].  Back to cited text no. 3
    
4.
Managers Manual on Community Monitoring of Health Services under NRHM. National Rural Health Mission (2005-2012). Ministry of Health and Family Welfare, Government of India. New Delhi; 2008.  Back to cited text no. 4
    
5.
Ruch-Ross H, Keller D, Miller N, Bassewitz J, Melinkovich P. Evaluation of community-based health projects: The healthy tomorrows experience. Pediatrics 2008;122:e564-72.  Back to cited text no. 5
    
6.
Kakde D. Compiled Report of Community Based Monitoring of Health Services under NRHM in Maharashtra (2007-10). Pune: SATHI State Nodal NGO for Community Based Monitoring of Health Services in Maharashtra; 2010.  Back to cited text no. 6
    
7.
Evaluation Study of National Rural Health Mission (NRHM) in 7 States. Programme Evaluation Organization Planning Commission, Government of India. New Delhi; 2011.  Back to cited text no. 7
    
8.
Singh R, Purohit B. Limitations in the functioning of village health and sanitation committees in a north western state in India. Int J Med Public Health 2012;2:39-46.  Back to cited text no. 8
  Medknow Journal  
9.
Mohan H, Jere D, Dodawad M, Ramaiah N, Kamati R, Garagatti S, et al. Village health and sanitation committees: Evaluation of a capacity-building intervention in Bagalkot and Koppal districts of northern Karnataka. BMC Proc 2012;6 Suppl 5:O24.  Back to cited text no. 9
    
10.
Butterfoss FD. Process evaluation for community participation. Annu Rev Public Health 2006;27:323-40.  Back to cited text no. 10
    
11.
Morgan LM. Community participation in health: Perpetual allure, persistent challenge. Health Policy Plan 2001;16:221-30.  Back to cited text no. 11
    
12.
Dutt KL, Samanta G. State initiatives for the empowerment of women of rural communities: Experiences from eastern India. Community Dev J 2002;37:137-56.  Back to cited text no. 12
    
13.
Macfarlane S, Racelis M, Muli-Musiime F. Public health in developing countries. Lancet 2000;356:841-6.  Back to cited text no. 13
    
14.
Oliveira-Cruz V, Hanson K, Mills A. Approaches to overcoming constraints to effective health service delivery: A review of the evidence. J Int Dev 2003;15:41-65.  Back to cited text no. 14
    


    Figures

  [Figure 1]
 
 
    Tables

  [Table 1], [Table 2], [Table 3], [Table 4]



 

Top
   
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
   Abstract
  Introduction
  Methodology
  Results
  Discussion
   Conclusion and R...
   References
   Article Figures
   Article Tables

 Article Access Statistics
    Viewed1322    
    Printed13    
    Emailed0    
    PDF Downloaded156    
    Comments [Add]    

Recommend this journal