World Rural Health Conference
Home Print this page Email this page Small font size Default font size Increase font size
Users Online: 1709
Home About us Editorial board Search Ahead of print Current issue Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents 
ORIGINAL ARTICLE
Year : 2019  |  Volume : 8  |  Issue : 7  |  Page : 2207-2211  

Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT)


1 AMCHSS (Achutha Menon Centre for Health Science Studies), Sree Chitra Tirunal Institute for Medical Sciences and Technology (SCTIMST), Trivandrum, Kerala, India
2 Ex-Department of Public Health and Preventive Medicine, Government of Tamil Nadu, Chennai, Tamil Nadu, India

Date of Submission10-May-2019
Date of Decision13-May-2019
Date of Acceptance22-May-2019
Date of Web Publication31-Jul-2019

Correspondence Address:
Dr. Upasana Sharma
Flat No. 97, F-1, First Floor, VGN Platina, Ambigai Nagar, Ayapakkam, Near Ambattur, Chennai - 600 077, Tamil Nadu
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jfmpc.jfmpc_380_19

Rights and Permissions
  Abstract 


Background: Risk assessment (RA) for mass gathering events is crucial to identify potential health hazards. It aids in planning and response activities specific to the event but is often overlooked by the event organizers. This paper reports the content validity process of a newly developed tool called Mass Gathering Risk Assessment Tool (MGRAT), which intends to assess the risks associated with religious mass gathering events in Indian settings. Methods: Qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. The draft tool was shared with six experts who were selected by the convenient method; selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool using content validity questionnaire. Content validity index and Fleiss kappa statistics were calculated to assess the agreement between multiple raters. Results: Agreement proportion expressed as scale-level content validity index (S-CVI) calculated by the averaging method is 0.92. S-CVI; calculated by universal agreement is 0.78. Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of the chance agreement is 0.522 (95% CI: 0.417, 0.628, P value: 0.001). Conclusion: MGRAT is a valid tool, which has an appropriate level of content validity. As the number of raters increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower Content Validity Index/Universal Average (CVI/UA) when compared with Content Validity Index/Average (CVI/Ave). Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also supports the appropriate content validity of MGRAT.

Keywords: Content validity, health risks, mass gathering, religious events, risk assessment


How to cite this article:
Sharma U, Desikachari B R, Sarma S. Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT). J Family Med Prim Care 2019;8:2207-11

How to cite this URL:
Sharma U, Desikachari B R, Sarma S. Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT). J Family Med Prim Care [serial online] 2019 [cited 2019 Aug 25];8:2207-11. Available from: http://www.jfmpc.com/text.asp?2019/8/7/2207/263793




  Introduction Top


According to the World Health Organization, MG event or MG is a gathering of persons that is usually defined as “the congregation of more than a specified number of people (this may be as few as one thousand persons; although most of the literature available describes these as gatherings that exceed 25,000 people) at a specific location for a specific purpose (a social function, a large public event, and a sports competition) for a defined period of time.”[1]

Mass gatherings cause notable challenges in terms of communicable and non-communicable disease surveillance, emergency preparedness, environmental health, vaccination, food safety, crowd management, etc.[2] In spite of the fact that MGs are an undeniably regular element of our society attended by huge crowds, yet such occasions are not very well understood. Although such gatherings are accumulations of “well people,” vast number of people associated with MGs can put a serious strain on the entire health care system.[3] Along these lines, such MG events are more perilous and hazardous in terms of higher incidence of injury and illness compared to the population in general.[4]

Preplanning for MG events is crucial and identification of potential health risks can be a vital element in the pre-event planning phase.[1] Risk assessment (RA) is an integral component of risk governance and serves as an initial step in the process of planning RA for an MG is a process that determines the intent and implementation of risk reduction measures, response planning, and capacity building for health functions.[5] RA for MGs is undertaken to empower the public health authorities to identify and evaluate the generic characteristics of a MG that introduce or escalate specific threats. Systematic assessment of risks also helps to identify the potential health security risks that require the cooperation of other departments and government agencies.[2]

During religious gatherings in India, some special events and unforeseen events occur at the places of religious MGs besides fixed places of worshipping. Special events such as idol procession, chariot pulling, fire walking, and animal sacrificing happen pulling larger crowds within the MG events and causing more damage to human beings and property. History is replete with incidences when MGs at fairs and festivals of India have turned into the hotspots of various types of risks and disasters.[6] A ten-year analysis of public health safety in 27 traditional MG events of India indicated around 936 dead and 540 injured casualties.[7]

Although public health system in India efficiently manages most of the religious MG events, there is no systematic process of the RA conducted in the field. Most of the process of RA is either overlooked or depends upon intuition and previous experiences. There is a need to systemize the process of RA for MG events for further risk reduction.[6],[8],[9] Hence, to place the right measures in place to address the foreseeable and unforeseeable risks, the proposed RA tool (MGRAT-Mass Gathering Risk Assessment Tool) in this study will be fundamental in identifying potential public health risks and prioritize planning and response activities specific to the religious MG events in an Indian setting.

Validity and reliability assessment are an essential part of tool development. Content validity is the index of whether the items in a test, both individually and as a whole represent the construct that it is proposed to measure. Content validity is concerned with the comprehensiveness of the item pool and representativeness of the mass gathering RA items included in the tool. After the initial design of the tool, content validity assessment is the leading process in the development of the tool to assess the contents for appropriateness for further field testing the tool.[10],[11],[12] This paper reports the content validity process of a newly developed tool called MGRAT, which intends to assess the risks associated with mass gathering events in the Indian settings.


  Methods Top


Initial process involved in the development of MGRAT tool

A qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. First, an extensive review of literature about the risks associated with the mass gatherings, theoretical basis for RA and available RA tools for mass gathering events was done. Second, key informants (n = 15) involved in the planning and management of religious mass gathering events in the state of Tamil Nadu, India were purposively identified and interviewed using a semi-structured interview guide. Principle of redundancy was followed. Content/Thematic analysis was done. A detailed explanation of the steps/process involved in identifying the domains and items is not reported in this paper.

Domains and Items identified

A sum total of forty-eight unique health risks were identified. Stampedes, fire accidents, structural collapse, drowning, outbreak of communicable diseases, exacerbation of existing medical illnesses (such as cardiac diseases, asthma, etc), etc., are some of the health risks identified. Seven domains (characteristics related to event, participant, environment, food and water related, disaster preparedness, medical service preparedness, and pre-event planning activities) and twenty-three items were generated from the content analysis of key informant interviews and literature review.

Content validity process

For evaluating content validity, the draft tool was shared with six experts who were selected by the convenient method. Draft tool was circulated among six subject experts for review (four experts from the department of public health and preventive medicine and two academics/research consultants). Selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool.

They were also requested to assess the tool according to whether all the items refer to relevant aspects of the construct to be measured, whether all the items together comprehensively reflect the construct and whether all the items are relevant for the setting where it is going to be applied, are simple and understandable. Using a self-administered content validity questionnaire, the experts were requested to assess the relevance of each item generated on a four-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant to avoid having a neutral and ambivalent midpoint).

Content validity questionnaire consisted of 28 questions. Out of these 23 questions addressed, the assessment of relevance of all the individual items (n = 23) under 7 domains in the developed tool. There were 5 questions addressed to assess the over-all relevance, comprehensiveness, usability, simplicity, understandability, etc.

Data collected from experts through content validity questionnaire were entered and analyzed with statistical software Epi info 7.1.5.2 version. Agreement proportions between the experts were calculated. Content validity index for overall scale (S-CVI) and item wise (I-CVI) were calculated. Fleiss kappa statistics to assess the agreement between multiple raters adjusting for chance agreement were calculated.

Ethical committee clearance

Approval for the study was obtained from the institutional ethics committee. Participants were informed about the purpose of the study by information sheet, which was provided in the English language and informed written consent for participation was obtained.


  Results Top


[Table 1] indicates that out of the twenty-three items, eighteen items were agreed upon as “Relevant” by all the six experts. The item “Solid waste management” was agreed upon as “Relevant” by only one rater and “Irrelevant” by the other five experts. Item-content validity index (I-CVI) of “Solid waste management” (item no 14) was calculated to be 0.17. The I-CVI of items “Participant origin (item no 7)” and “Psychosocial behavior of participants (item no 9)” was 0.67 and I-CVI of item “Area involved (item no 4)” and “Fire safety (item no 20)” was 0.83. The items 7, 9, 14, and 20 were retained in the tool with minor modifications, and “Solid waste management” (item no 14) was merged with “Sanitary and hygiene facilities.”
Table 1: Content Validity Index (CVI)

Click here to view


Agreement proportion expressed as scale level content validity index (S-CVI) calculated by averaging method (S-CVI/Ave = average of all I-CVI) was calculated to be 0.92. In addition, S-CVI calculated by universal agreement method (S-CVI/UA = Total number of items agreed relevant by all six experts/Total number of items) was calculated to be 0.78. [Table 2] shows Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement was 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss' kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement.
Table 2: Fleiss Kappa statistics to assess agreement between multiple experts (n=6)

Click here to view



  Discussion Top


The study reports the content validity process of the newly developed tool called MGRAT. The tool was shared with six subject experts, and they were requested to assess and rate the tool. Agreement between the experts expressed as CVI and Fleiss kappa statistics in assessing the tool as “Relevant” was calculated and reported.

Agreement proportion measured by averaging method showed 92% agreement and when measured by universal agreement method, it showed 78% agreement. These results support the excellent content validity of the developed MGRAT. However, the CVI of the instrument using universal agreement approach was found to be low compared to the averaging method. This is because of the fact that high number of content experts makes consensus difficult. The developed tool MGRAT is a pioneer attempt to develop a RA tool to assess health risks associated with mass gathering events in Indian settings. We could not find previously reported measures of agreement to compare with the agreement values of MGRAT found in this study. However, content validity studies reported that scale with excellent content validity should be composed of I-CVIs of 0.78 or higher value and S-CVI/UA and S-CVI/Ave of 0.7 and 0.9 or higher value, respectively.[11],[13],[14],[15]

Although content validity index measures are widely used to estimate content validity by researchers, the problems with those measures are that they overlook the percentage of agreement between the experts owing to chance. Therefore, Wynd et al., propose that both CVI and multi-rater kappa statistic should be calculated in the content validity studies because unlike the CVI, kappa adjusts for chance agreement.[12] The chance agreement is an issue of concern while studying agreement indices among assessors, especially when we place four-point scoring within two “relevant” and “not relevant” classes. In other words, kappa statistic is a consensus index of inter-rater agreement that adjusts for chance agreement and is an important supplement to CVI because Kappa provides information about the degree of agreement beyond chance. Nevertheless, the CVI is mostly used by researchers because it is simple for calculation, easy to understand, and provide information about each item, which can be used for modification or deletion of instrument items.[10],[12]

In the present study, Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement showed a significant agreement value of 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss' kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement. Overall, CVI and Fleiss' kappa (κ) both the measures suggest that the content validity of the newly developed tool is appropriate. MGRAT was judged to be valid, appropriate, and feasible to assess the health risks associated with the MG events.

All seven domains were retained after the content validity process. Out of 23 items, 22 items were retained and 1 item was removed after the content validity process. The item “Solid waste management” was merged with “Sanitation and Hygiene facilities” after the content validity process as it was agreed upon as relevant by only one rater and irrelevant by the other five experts. Thus, in the refined MGRAT after content validity, there were 7 domains and 22 items. This validated MGRAT was developed further into a mobile web APP (web-based application) and field tested for its feasibility assessment in one of the religious mass gathering events in Tamil Nadu, India.


  Limitations Top


Limitations of the study are that the experts' feedback is subjective; thus, the study is subjected to bias that may exist among the experts. If content domain is not well-identified, this type of study does not necessarily identify content that might have been omitted from the instrument owing to the assessment. However, experts are asked to suggest other items for the instrument, which might help minimize this limitation.

This content validity study included a multidisciplinary team of experts. An effort was made to quantify the agreement and efforts to report kappa statistics to assess agreement beyond the chance agreement are some of the advantages of this study. Added to that, the idea of the development of systematic RA tool to assess health risks associated with mass gathering events is itself pioneer effort in India.

Content validity is a systematic process, which includes the judgment/quantification on instrument items by content experts. Such a process should be the leading study in the process of making an instrument to guarantee instrument reliability and prepare a valid instrument in terms of content for the preliminary test phase.[16] To conclude, the present study indicated that the MGRAT is a valid tool, which enjoys an appropriate level of content validity. As the number of rater's increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower CVI/UA when compared with CVI/Ave. Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also support the appropriate content validity of MGRAT.

Acknowledgements

We thank Dr. Manoj Murhekar (Director, ICMR-NIE, Chennai), Dr. Sanjay M Mehendale (Ex-ADG, ICMR, New Delhi) and Dr. R Ramakrisnan (Ex-Scientist -G, ICMR-NIE, Chennai) for their support.

Financial support and sponsorship

Upasana Sharma is supported by Indian Council of Medical Research (ICMR) Senior Research Fellowship.

Conflicts of interest

There are no conflicts of interest.



 
  References Top

1.
World Health Organization. Public health for mass gatherings: Key considerations. 2015.  Back to cited text no. 1
    
2.
Tam JS, Barbeschi M, Shapovalova N, Briand S, Memish ZA, Kieny M-P. Research agenda for mass gatherings: A call to action. Lancet Infect Dis 2012;12:231-9.  Back to cited text no. 2
    
3.
Memish ZA, Al-Rabeeah AA. Public health management of mass gatherings: The Saudi Arabian experience with MERS-CoV. Bull World Health Organ 2013;91:899-899A.  Back to cited text no. 3
    
4.
Arbon P. The development of conceptual models for mass-gathering health. Prehosp Disaster Med 2004;19:208-12.  Back to cited text no. 4
    
5.
Fleischauer AT, Gaines J. Enhancing surveillance for mass gatherings: The role of syndromic surveillance. Public Health Rep 2017;132 (1 Suppl):95S-8S.  Back to cited text no. 5
    
6.
Illiyas FT, Mani SK, Pradeepkumar AP, Mohan K. Human stampedes during religious festivals: A comparative review of mass gathering emergencies in India. Int J Disaster Risk Reduct 2013;5:10-8.  Back to cited text no. 6
    
7.
Sanyal S, Madan A. (P2-39) Public health safety for traditional mass gatherings in India: A 10-year analysis. Prehosp Disaster Med 2011;26(S1):s(156).  Back to cited text no. 7
    
8.
Greenough PG. The KumbhMela stampede: Disaster preparedness must bridge jurisdictions. BMJ 2013;346:f3254.  Back to cited text no. 8
    
9.
A comprehensive review of the KumbhMela: Identifying risks for spread of infectious diseases-ScienceDirect [Internet]. [cited 2019 May 9]. Available from: https://www.sciencedirect.com/science/article/pii/S1198743X14001074.  Back to cited text no. 9
    
10.
Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382-6.  Back to cited text no. 10
    
11.
Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res 2003;27:94-104.  Back to cited text no. 11
    
12.
Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res 2003;25:508-18.  Back to cited text no. 12
    
13.
Shi J, Mo X, Sun Z. [Content validity index in scale development]. Zhong Nan Da Xue Xue Bao Yi Xue Ban 2012;37:152-5.  Back to cited text no. 13
    
14.
Polit DF, Beck CT. The content validity index: Are you sure you know what's being reported? Critique and recommendations. Res Nurs Health 2006;29:489-97.  Back to cited text no. 14
    
15.
Blais MA, Hilsenroth MJ, Castlebury FD. Content validity of the dsm-iv borderline and narcissistic personality disorder criteria sets. Compr Psychiatry 1997;38:31-7.  Back to cited text no. 15
    
16.
Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar A-R. Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. J Caring Sci 2015;4:165-78.  Back to cited text no. 16
    



 
 
    Tables

  [Table 1], [Table 2]



 

Top
   
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
   Abstract
  Introduction
  Methods
  Results
  Discussion
  Limitations
   References
   Article Tables

 Article Access Statistics
    Viewed60    
    Printed0    
    Emailed0    
    PDF Downloaded12    
    Comments [Add]    

Recommend this journal