|Year : 2015 | Volume
| Issue : 4 | Page : 539-545
Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges
Jaya Prasad Tripathy1, Arun Kumar Aggarwal2, Binod Kumar Patro3, Himbala Verma4
1 Department of Community Medicine, School of Public Health, PGIMER, Chandigarh; International Union Against Tuberculosis and Lung Disease, The Union South East Asia Office, New Delhi, India
2 Department of Community Medicine, School of Public Health, PGIMER, Chandigarh, India
3 Department of Community Medicine and Family Medicine, AIIMS, Bhubaneswar, Odisha, India
4 Public Health Consultant, National Health Mission, Chandigarh, India
|Date of Web Publication||18-Jan-2016|
Jaya Prasad Tripathy
Department of Community Medicine, School of Public Health, PGIMER, Chandigarh
Source of Support: None, Conflict of Interest: None
Background: Community monitoring was introduced on a pilot mode in 36 selected districts of India in a phased manner. In Chandigarh, it was introduced in the year 2009-2010. A preliminary evaluation of the program was undertaken with special emphasis on the inputs and the processes. Methodology: Quantitative methods included verification against checklists and record reviews. Nonparticipant observation was used to evaluate the conduct of trainings, interviews, and group discussions. Health system had trained health system functionaries (nursing students and Village Health Sanitation Committee [VHSC] members) to generate village-based scorecards for assessing community needs. Community needs were assessed independently for two villages under the study area to validate the scores generated by the health system. Results: VHSCs were formed in all 22 villages but without a chairperson or convener. The involvement of VHSC members in the community monitoring process was minimal. The conduct of group discussions was below par due to poor moderation and unequal responses from the group. The community monitoring committees at the state level had limited representation from the non-health sector, lower committees, and the nongovernmental organizations/civil societies. Agreement between the report cards generated by the investigator and the health system in the selected villages was found to be to be fair (0.369) whereas weighted kappa (0.504) was moderate. Conclusion: In spite of all these limitations and challenges, the government has taken a valiant step by trying to involve the community in the monitoring of health services. The dynamic nature of the community warrants incorporation of an evaluation framework into the planning of such programs.
Keywords: Chandigarh, community monitoring, process evaluation
|How to cite this article:|
Tripathy JP, Aggarwal AK, Patro BK, Verma H. Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges
. J Family Med Prim Care 2015;4:539-45
|How to cite this URL:|
Tripathy JP, Aggarwal AK, Patro BK, Verma H. Process evaluation of community monitoring under national health mission at Chandigarh, union territory: Methodology and challenges
. J Family Med Prim Care [serial online] 2015 [cited 2019 Nov 21];4:539-45. Available from: http://www.jfmpc.com/text.asp?2015/4/4/539/174282
| Introduction|| |
The Alma Ata Declaration  (1978), Ottawa Charter  (1986), and Agenda 21  (1992) among others have placed community participation high on the political and public health agendas of member nations. National Rural Health Mission (NRHM), Chandigarh, India also spells clearly the importance of community participation as part of the decentralized process of health care management. 
Community Monitoring under NRHM was implemented in a phased manner in India. In Chandigarh, it was introduced in the year 2009-2010. It empowers the community to work with the health service providers to identify and solve the community problems. Government of India (GOI) has issued comprehensive guidelines for the same.  States were supposed to adopt the guidelines keeping the spirit of community representation intact.
All implementing states should review this novel but challenging strategy periodically. However, the methodology, tools, and experiences for its evaluation are very limited. Thus, we undertook a process evaluation of the community monitoring implementation in Chandigarh with the following objectives: (a) How well organizational structures for community monitoring were set up (b) How were trainings organized and how much knowledge and skill improvements occurred with the trainings (c) to what extent communities participated in the problem assessments and had discussion with the health service providers (d) to understand the gaps in implementation, and (e) to ascertain the feasibility of the methods and tools used to measure implementation of community monitoring.
| Methodology|| |
The study was carried out in the union territory of Chandigarh. Mixed methodology approach was followed. Both quantitative and qualitative methods were used for evaluation. Methods such as desk review, in-depth interviews, checklists, nonparticipant observation, training evaluation, etc., were employed to assess various components of community monitoring implementation. The major structural and functional domains of the community monitoring process were evaluated, that is, organizational set up, quality of trainings, conduct of interviews and group discussions, and the involvement of the community in the entire exercise.
Organizational set up
A structured scoring checklist was developed to evaluate the monitoring committees at two levels, that is, state/union territory and village level namely the State Monitoring and Planning Committee, State Mentoring Committee and the Village Health and Sanitation Committee (VHSC). In-depth, interviews were conducted with the concerned nodal officer and records were reviewed to understand the constitution of the committees.
Quality of trainings
Records were reviewed to evaluate the quality of trainings viz., number and profile of participants, training methodology, and feedback. A checklist was developed for scrutiny of the training records. As very minimal record was available, it was not possible to comment on the quality of the training. However, during the course of the study, another training was scheduled which was evaluated by one of the investigators (JPT) who attended the training as a nonparticipant observer. Observations were recorded using a scoring checklist.
To assess the impact of the training, knowledge of the health system staff (nursing students and VHSC members) was assessed using a structured questionnaire. It had questions on entitlements under NRHM and preparation of report cards. A comparison was also made between the levels of knowledge between the trained (93) and the untrained (43) health system functionaries.
Conduct of the interviews and group discussions
The process of data collection viz., interviews with maternal beneficiaries and group discussions were evaluated by observation in two villages using a scoring checklist. The investigator (JPT) undertook nonparticipant observation using a checklist to observe the interviews and focus group discussions (FGDs) in these villages.
Validation of report card scores
Investigator (JPT) made independent visit to some houses and filled the scorecards himself to validate the findings with those filled by the health system staff who were trained for the same by the union territory health department. Investigator visited the houses about 15 days after the visit of the trained health staff. These scorecards are color coded with red, yellow, and green color codes signifying the urgency of the action, with red being most urgent and green as normal.
Kappa agreement between the color codes of the report cards generated by the investigator and the trained health staff in the same villages was also calculated.
Flow of information
This was assessed by review of records, scrutiny of village health report cards, and discussions with the program managers.
The study was ethically approved by the Institute Ethics Committee, PostGraduate Institute of Medical Education and Research, Chandigarh, India.
| Results|| |
Status of setting up of committees at state and village level as per Government of India guidelines
For setting up of state/union territory level committees as per the GOI guidelines, union territory Chandigarh achieved a score of 7 marks out of 20 (35%). State Monitoring and Planning Committee had no or minimal representation from the elected bodies, lower committees, civil society, or other departments [Table 1]. The mentoring committee at union territory level comprised of only three members, all of them being faculty members in the Department of Community Medicine in two medical colleges in the state. Thus, there was no representation from the state health department, nodal nongovernmental organization (NGO) as well from the civil society in the mentoring committee.
|Table 1: Score sheet for organizational set up in Chandigarh under community monitoring |
Click here to view
VHSCs are to be constituted at the village level for village level planning, which is a crucial step toward accomplishing the goal of community monitoring. Chandigarh has 22 villages under its jurisdiction. All the 22 villages in Chandigarh have a VHSC. Each VHSC in a village was assessed based on eight parameters. Of a total score of 176, the villages scored 100 (57%) [Table 2]. None of the VHSCs has a nominated chairperson and a convener. The majority (144, 84.2%) of the members are females. Auxiliary nurse midwife (ANM) is present in all VHSCs, and Anganwadi worker (AWW) is present in 21 (95%) of 22 villages. VHSCs in villages of Chandigarh have doctors and social workers in 18 villages (82%). Panch is a member of VHSC in 17 villages (77%). Half of the villages have SC/ST candidate in the committee whereas only 9 villages (41%) have a teacher in their committees.
|Table 2: Score sheet for evaluating organizational set up at the village level |
Click here to view
Quality of trainings and assessment of knowledge assessment of the trained health system functionaries
Training record reviews did not yield desired information about the quality of sessions, teaching methods used, training curriculum, and the training coordinator. However, participant observation of a training program revealed that training was done in good ambience. However, trainings were lecture-oriented with all the participants attending the training together giving a high trainer: Trainee ratio of 1:70. No skills were imparted for conduct of interviews and organization of FGDs. The training session on tools of monitoring and data collection were taken hurriedly with little or no discussion.
A total of 93 health system functionaries (nursing students), who had received the training, were administered a knowledge assessment questionnaire. Seventy-two percent (67/93) secured at least 50% marks on questions related to health entitlements under NRHM whereas only 51 (55%) students could score at least 50% marks when enquired about the Preparation of Report Cards. Sixty-two percent (58/93) students scored at least 50% marks overall . Questionnaire was also administered to 69 VHSC members. None of them scored at least 50% marks in overall knowledge and the preparation of report card section. However, 61% (42/69) VHSC members interviewed scored more than 50% marks regarding Health Entitlements under NRHM [Table 3].
|Table 3: Comparison of knowledge scores among different categories of health system functionaries |
Click here to view
Forty-three untrained staff (nursing students) of the same batch were also administered the same questionnaire. There was significant difference between the mean scores of trained staff compared to the untrained ones regarding preparation of report cards and the overall knowledge gained; for the questions on health entitlements under NRHM, no significant difference was seen in the mean scores of the two groups [Table 3].
The group discussions achieved a score of 50% (5/10) whereas the interviews scored 60% (6/10). The conduct of the group discussions was dismal with poor moderation and unequal responses from the group. The group discussions were more of a lecturing rather than being interactive.
The percentage of scores obtained by both categories of health system staff namely nursing students and VHSC members regarding preparation of report cards were 50% and 9.3%, respectively (P < 0.001). The difference in the scores about knowledge regarding health entitlements under NRHM was found to be insignificant. However, the overall mean score was found to significantly different between the two groups (P < 0.001) [Table 3]. Among VHSC members, the health-related members namely the ANM, AWW, and the teachers among the nonhealth members fared better than the other members.
Flow of information
The management information system in Chandigarh witnessed a deviation from the standard guidelines. The information after being collected at the village level goes directly to the state bypassing the VHSC. Thus, we have two models of MIS now (in the context of Chandigarh) as given in the figure [Figure 1].
A team of three consultants that is, two Public Health consultants from NRHM and faculty from the Department of Community Medicine, GMCH-32, Chandigarh visited all the villages and held meetings with the members of VHSC in the respective villages. They discussed major issues such as biomedical waste management, absence of doctor in the dispensary, utilization of VHSC and untied funds, JSY irregularities, poor health infrastructure, etc., Concrete remedial steps were taken to resolve some of these issues like opening of VHSC account, training to VHSC members regarding utilization of VHSC and untied funds, appointment of doctors. However, the administration was found wanting in issues like biomedical waste management and Poor infrastructure. Resolution of such issues needs a strong inter-sectoral coordinating framework which is grossly lacking in the organizational framework under Community Monitoring in Chandigarh.
Validation of the report cards
Investigator (JPT) independently visited some houses and filled the report cards. Kappa agreement between the color codes in the report cards generated by the investigator and the trained health staff was found to be to be fair (0.369) whereas weighted kappa (0.504) was moderate [Table 4].
|Table 4: 3×3 table showing the agreement between the health system functionaries and the investigator |
Click here to view
| Discussion|| |
Evaluation of community-based activities is necessary for continuous improvement and sustainability of the program.  Community monitoring is an important strategy under NRHM. However, there are not much experiences available in the indexed literature about the methods and tools to monitor the implementation. In our study, we developed methods and tools and applied the same to evaluate the process of community monitoring. There were some important findings that could be useful to the program implementers and future researchers.
Involvement of community representatives and NGOs in committees at various levels was grossly missing which defeats the prime objective of the program. This may be due to the common notion among health officials that direct involvement of community can put the entire system under stress.
The organizational set up in Chandigarh is a two-tier level as against the five-tier hierarchical structure proposed in the national guidelines. However, the unique administrative structure in Chandigarh well justifies the two-tier system. The State Level Monitoring and Planning Committee has little or no representation from the elected bodies, committees at lower levels, (i.e. VHSCs) nonhealth departments and the nodal NGO/civil society involved in community monitoring. Poor coordinating mechanism with other sectors and noninvolvement of NGOs and civil societies hinders effective decision making and implementing ability.
Maharashtra has been a successful model so far with adequate representation from the lower committee, NGO, health department, nonhealth sectors, and the elected ones in the committees at all levels.  The three-membered mentoring committee in Chandigarh, all of them being faculty members of medical colleges in Chandigarh, is in sharp contrast with that of Maharashtra which has 15 members, 11 eminent members from civil society organizations experienced in community-based work.
Among the different innovations tried, the recruitment of nursing students as the facilitating NGO was certainly one of them. The nursing students come from a health-related background and also there is little variation in the educational status, age and other personal attributes. Thus, it might be more effective to train such students rather than training people from diverse backgrounds mostly nonhealth.
Quality of trainings
Training quality assessment through participant observation proved very useful to assess the strengths and the gaps. Trainings were mostly lecture-oriented with no focus on skill building for interviews and FGDs. This got reflected in the poor conduct of interviews and FGDs in the community. The training should focus on community engagement, tools of community monitoring, preparation of report cards, conducting village health meetings, and group discussions. Field-based practical hands-on training and role plays might be useful.
Quality of group discussions
The quality of the group discussions observed was quite poor. They had females in the majority, and they were mostly the patients coming to the health facility or the AWW. This might confound the responses. The group discussions were more of a one-to-one lecturing rather than being an interactive discussion. Active participation from all the participants in a group discussion could not be elicited due to poor moderation. This necessitates field training or role play as part of training curriculum regarding conduct of interviews and group discussions.
Composition and functioning of village health sanitation committees
VHSC meetings were not being held regularly, and attendance of the meetings was also found to be poor. This might be due to the fact that the VHSCs in Chandigarh do not have a designated Chairperson and a convener. The uncertainty in the roles of the members of the committees was probably one of the reasons for the infrequent meetings of VHSCs. Doctors were observed to be the members of the village committees in 18 of 22 villages. Such precedence in VHSCs is not found in other parts of the country.
The NRHM evaluation study by Planning Commission in seven states also reiterates the fact that barring Tamil Nadu VHSCs are not discharging their duties as one would have liked to.  Another study in North India found poor awareness among the VHSC members about functions, members and activities of the committee and areas of utilization of the VHSC fund.  The Chandigarh health department is striving hard to strengthen the role of VHSCs in village health planning and action. The VHSC members have been trained recently about the preparation of village health plan, utilization of untied and VHSC funds, organizing regular meetings, and preparing meeting reports.
The poor knowledge of the VHSC members especially the nonhealth members regarding their roles and responsibilities and the preparation of report cards warrants a capacity building exercise to empower them to ensure their participation and guarantee quality health care services.
Involvement of the community
Engaging the community in planning and monitoring of health service delivery is central to enhancing the availability, accessibility, quality, and use of the public health system.  However, community monitoring in Chandigarh is working without its soul, that is, the community. The involvement of the community in this entire process is poor. Maharashtra through a couple of novel innovations tried to involve the community in the preparation of report cards. They converted the report card into a poster and put up in several places in the village so that everybody could participate in report card preparation process. Another innovation was the Arogya Jagruti Diwas, a long-day program where people used to gather in at a place and participate in the report card preparation process. 
Village Health Meetings and Jan Sunwais are forums where the report cards are shared with the community, and people share their experiences of seeking health care. These meetings actually ensure involvement of the user group or the general community. A diverse range of issues have been raised during Jan Sunwais, including: Availability of medicines, ambulance services, irregularities in the provision of incentives, corruption and illegal charging, attitude of the service providers, instances of denial of health services, etc., in the states of Maharashtra, Karnataka, and Rajasthan as part of community monitoring. But such community intensive activities could not be organized in any of the 22 villages in union territory, Chandigarh. Thus, the collated report cards, instances of denial of health care, and other local issues and grievances were never discussed in a public forum thereby failing to involve the community in the process.
Weighted kappa revealed moderate agreement between the report card scores compiled by the trained health staff and the investigator himself. The domains which showed good agreement were maternal health, child health, equipment, infrastructure, etc., while JSY payment, quality of care, unofficial charges, and disease surveillance yielded poor agreement (red when interviewed by the trained staff and yellow by the investigator). Probably the respondents failed to express themselves in front of the investigator but they reported grievances before the trained health staff. There can be some plausible reasons for this disagreement. The 15-day time gap between the data collection by the investigator and the trained health staff might have witnessed a withdrawal of a certain health care service which might have bred discontent among the respondents but that looks very unlikely. The maternal beneficiaries interviewed and the composition of the group discussions were different in both the cases which might have elicited different responses although we can argue that the community from which all the respondents have come is the same and have the same set of issues. Gender difference might also play a determining role in eliciting true responses. A group of female moderators/interviewers is more likely to generate true responses than a single male investigator particularly when the respondent is a female which is usually the case here. Thus, a team of two females moderating a group discussion at an Anganwadi center looks a better proposition. These domains are such that they require good rapport building with the community to extract true responses. This disagreement can be tackled if there are user group representatives actively involved in the process of data collection.
The tools and methods used in this study were similar to those suggested by Butterfoss who also mentioned methods such as participant surveys, key informant interviews, FGDs, observation of meetings, and review of documents to evaluate community-based studies.  Similar methodology was also used by another.
Studies of community participation show promising short-term results but have mixed record of long-term success. This evaluation was done just 2 years after the implementation of the program. Thus, it being the initial phase might elicit the better response from the community. Moreover, structural, economic, cultural, and social constraints such as poverty, caste, class and gender hierarchies, lack of knowledge, access to health care, and weak health system hinder community participation. ,,, These factors could not be assessed in the present study.
| Conclusion and Recommendations|| |
Early and long-term participation by community members is needed to sustain community-based projects. The dynamic nature of such programs warrants the incorporation of an evaluation framework into the planning of such programs.  Community-based monitoring in Chandigarh failed up to its expectations when it came down to effective engagement of the community. Monitoring and Planning Committees should include representation from NGOs/civil society and lower committees such as VHSCs. Nonhealth VHSC members and other key persons of the village to be involved in the process of data collection to promote greater involvement of the community and other stakeholders. Operations research needs to be carried out to make the tools simpler, understandable and contextual. Practical hands-on field training and role plays should be the part of the training curriculum with more focus on community engagement, organizing and conducting group discussions, and Jan Sunwais.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
WHO. Ottawa Charter for Health Promotion. Geneva: World Health Organization; 1986.
Managers Manual on Community Monitoring of Health Services under NRHM. National Rural Health Mission (2005-2012). Ministry of Health and Family Welfare, Government of India. New Delhi; 2008.
Ruch-Ross H, Keller D, Miller N, Bassewitz J, Melinkovich P. Evaluation of community-based health projects: The healthy tomorrows experience. Pediatrics 2008;122:e564-72.
Kakde D. Compiled Report of Community Based Monitoring of Health Services under NRHM in Maharashtra (2007-10). Pune: SATHI State Nodal NGO for Community Based Monitoring of Health Services in Maharashtra; 2010.
Evaluation Study of National Rural Health Mission (NRHM) in 7 States. Programme Evaluation Organization Planning Commission, Government of India. New Delhi; 2011.
Singh R, Purohit B. Limitations in the functioning of village health and sanitation committees in a north western state in India. Int J Med Public Health 2012;2:39-46.
Mohan H, Jere D, Dodawad M, Ramaiah N, Kamati R, Garagatti S, et al
. Village health and sanitation committees: Evaluation of a capacity-building intervention in Bagalkot and Koppal districts of northern Karnataka. BMC Proc 2012;6 Suppl 5:O24.
Butterfoss FD. Process evaluation for community participation. Annu Rev Public Health 2006;27:323-40.
Morgan LM. Community participation in health: Perpetual allure, persistent challenge. Health Policy Plan 2001;16:221-30.
Dutt KL, Samanta G. State initiatives for the empowerment of women of rural communities: Experiences from eastern India. Community Dev J 2002;37:137-56.
Macfarlane S, Racelis M, Muli-Musiime F. Public health in developing countries. Lancet 2000;356:841-6.
Oliveira-Cruz V, Hanson K, Mills A. Approaches to overcoming constraints to effective health service delivery: A review of the evidence. J Int Dev 2003;15:41-65.
[Table 1], [Table 2], [Table 3], [Table 4]