|Year : 2019 | Volume
| Issue : 7 | Page : 2207-2211
Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT)
Upasana Sharma1, BR Desikachari2, Sankara Sarma1
1 AMCHSS (Achutha Menon Centre for Health Science Studies), Sree Chitra Tirunal Institute for Medical Sciences and Technology (SCTIMST), Trivandrum, Kerala, India
2 Ex-Department of Public Health and Preventive Medicine, Government of Tamil Nadu, Chennai, Tamil Nadu, India
|Date of Submission||10-May-2019|
|Date of Decision||13-May-2019|
|Date of Acceptance||22-May-2019|
|Date of Web Publication||31-Jul-2019|
Dr. Upasana Sharma
Flat No. 97, F-1, First Floor, VGN Platina, Ambigai Nagar, Ayapakkam, Near Ambattur, Chennai - 600 077, Tamil Nadu
Source of Support: None, Conflict of Interest: None
Background: Risk assessment (RA) for mass gathering events is crucial to identify potential health hazards. It aids in planning and response activities specific to the event but is often overlooked by the event organizers. This paper reports the content validity process of a newly developed tool called Mass Gathering Risk Assessment Tool (MGRAT), which intends to assess the risks associated with religious mass gathering events in Indian settings. Methods: Qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. The draft tool was shared with six experts who were selected by the convenient method; selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool using content validity questionnaire. Content validity index and Fleiss kappa statistics were calculated to assess the agreement between multiple raters. Results: Agreement proportion expressed as scale-level content validity index (S-CVI) calculated by the averaging method is 0.92. S-CVI; calculated by universal agreement is 0.78. Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of the chance agreement is 0.522 (95% CI: 0.417, 0.628, P value: 0.001). Conclusion: MGRAT is a valid tool, which has an appropriate level of content validity. As the number of raters increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower Content Validity Index/Universal Average (CVI/UA) when compared with Content Validity Index/Average (CVI/Ave). Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also supports the appropriate content validity of MGRAT.
Keywords: Content validity, health risks, mass gathering, religious events, risk assessment
|How to cite this article:|
Sharma U, Desikachari B R, Sarma S. Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT). J Family Med Prim Care 2019;8:2207-11
|How to cite this URL:|
Sharma U, Desikachari B R, Sarma S. Content validity of the newly developed risk assessment tool for religious mass gathering events in an Indian setting (Mass Gathering Risk Assessment Tool-MGRAT). J Family Med Prim Care [serial online] 2019 [cited 2019 Dec 12];8:2207-11. Available from: http://www.jfmpc.com/text.asp?2019/8/7/2207/263793
| Introduction|| |
According to the World Health Organization, MG event or MG is a gathering of persons that is usually defined as “the congregation of more than a specified number of people (this may be as few as one thousand persons; although most of the literature available describes these as gatherings that exceed 25,000 people) at a specific location for a specific purpose (a social function, a large public event, and a sports competition) for a defined period of time.”
Mass gatherings cause notable challenges in terms of communicable and non-communicable disease surveillance, emergency preparedness, environmental health, vaccination, food safety, crowd management, etc. In spite of the fact that MGs are an undeniably regular element of our society attended by huge crowds, yet such occasions are not very well understood. Although such gatherings are accumulations of “well people,” vast number of people associated with MGs can put a serious strain on the entire health care system. Along these lines, such MG events are more perilous and hazardous in terms of higher incidence of injury and illness compared to the population in general.
Preplanning for MG events is crucial and identification of potential health risks can be a vital element in the pre-event planning phase. Risk assessment (RA) is an integral component of risk governance and serves as an initial step in the process of planning RA for an MG is a process that determines the intent and implementation of risk reduction measures, response planning, and capacity building for health functions. RA for MGs is undertaken to empower the public health authorities to identify and evaluate the generic characteristics of a MG that introduce or escalate specific threats. Systematic assessment of risks also helps to identify the potential health security risks that require the cooperation of other departments and government agencies.
During religious gatherings in India, some special events and unforeseen events occur at the places of religious MGs besides fixed places of worshipping. Special events such as idol procession, chariot pulling, fire walking, and animal sacrificing happen pulling larger crowds within the MG events and causing more damage to human beings and property. History is replete with incidences when MGs at fairs and festivals of India have turned into the hotspots of various types of risks and disasters. A ten-year analysis of public health safety in 27 traditional MG events of India indicated around 936 dead and 540 injured casualties.
Although public health system in India efficiently manages most of the religious MG events, there is no systematic process of the RA conducted in the field. Most of the process of RA is either overlooked or depends upon intuition and previous experiences. There is a need to systemize the process of RA for MG events for further risk reduction.,, Hence, to place the right measures in place to address the foreseeable and unforeseeable risks, the proposed RA tool (MGRAT-Mass Gathering Risk Assessment Tool) in this study will be fundamental in identifying potential public health risks and prioritize planning and response activities specific to the religious MG events in an Indian setting.
Validity and reliability assessment are an essential part of tool development. Content validity is the index of whether the items in a test, both individually and as a whole represent the construct that it is proposed to measure. Content validity is concerned with the comprehensiveness of the item pool and representativeness of the mass gathering RA items included in the tool. After the initial design of the tool, content validity assessment is the leading process in the development of the tool to assess the contents for appropriateness for further field testing the tool.,, This paper reports the content validity process of a newly developed tool called MGRAT, which intends to assess the risks associated with mass gathering events in the Indian settings.
| Methods|| |
Initial process involved in the development of MGRAT tool
A qualitative approach was followed to identify the risks associated with mass gathering events and to identify the domains and items to be included in the RA tool. First, an extensive review of literature about the risks associated with the mass gatherings, theoretical basis for RA and available RA tools for mass gathering events was done. Second, key informants (n = 15) involved in the planning and management of religious mass gathering events in the state of Tamil Nadu, India were purposively identified and interviewed using a semi-structured interview guide. Principle of redundancy was followed. Content/Thematic analysis was done. A detailed explanation of the steps/process involved in identifying the domains and items is not reported in this paper.
Domains and Items identified
A sum total of forty-eight unique health risks were identified. Stampedes, fire accidents, structural collapse, drowning, outbreak of communicable diseases, exacerbation of existing medical illnesses (such as cardiac diseases, asthma, etc), etc., are some of the health risks identified. Seven domains (characteristics related to event, participant, environment, food and water related, disaster preparedness, medical service preparedness, and pre-event planning activities) and twenty-three items were generated from the content analysis of key informant interviews and literature review.
Content validity process
For evaluating content validity, the draft tool was shared with six experts who were selected by the convenient method. Draft tool was circulated among six subject experts for review (four experts from the department of public health and preventive medicine and two academics/research consultants). Selected experts were requested to assess the tool and give their comments about the domains, items, relevant responses, and overall presentation of the tool.
They were also requested to assess the tool according to whether all the items refer to relevant aspects of the construct to be measured, whether all the items together comprehensively reflect the construct and whether all the items are relevant for the setting where it is going to be applied, are simple and understandable. Using a self-administered content validity questionnaire, the experts were requested to assess the relevance of each item generated on a four-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant to avoid having a neutral and ambivalent midpoint).
Content validity questionnaire consisted of 28 questions. Out of these 23 questions addressed, the assessment of relevance of all the individual items (n = 23) under 7 domains in the developed tool. There were 5 questions addressed to assess the over-all relevance, comprehensiveness, usability, simplicity, understandability, etc.
Data collected from experts through content validity questionnaire were entered and analyzed with statistical software Epi info 18.104.22.168 version. Agreement proportions between the experts were calculated. Content validity index for overall scale (S-CVI) and item wise (I-CVI) were calculated. Fleiss kappa statistics to assess the agreement between multiple raters adjusting for chance agreement were calculated.
Ethical committee clearance
Approval for the study was obtained from the institutional ethics committee. Participants were informed about the purpose of the study by information sheet, which was provided in the English language and informed written consent for participation was obtained.
| Results|| |
[Table 1] indicates that out of the twenty-three items, eighteen items were agreed upon as “Relevant” by all the six experts. The item “Solid waste management” was agreed upon as “Relevant” by only one rater and “Irrelevant” by the other five experts. Item-content validity index (I-CVI) of “Solid waste management” (item no 14) was calculated to be 0.17. The I-CVI of items “Participant origin (item no 7)” and “Psychosocial behavior of participants (item no 9)” was 0.67 and I-CVI of item “Area involved (item no 4)” and “Fire safety (item no 20)” was 0.83. The items 7, 9, 14, and 20 were retained in the tool with minor modifications, and “Solid waste management” (item no 14) was merged with “Sanitary and hygiene facilities.”
Agreement proportion expressed as scale level content validity index (S-CVI) calculated by averaging method (S-CVI/Ave = average of all I-CVI) was calculated to be 0.92. In addition, S-CVI calculated by universal agreement method (S-CVI/UA = Total number of items agreed relevant by all six experts/Total number of items) was calculated to be 0.78. [Table 2] shows Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement was 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss' kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement.
|Table 2: Fleiss Kappa statistics to assess agreement between multiple experts (n=6)|
Click here to view
| Discussion|| |
The study reports the content validity process of the newly developed tool called MGRAT. The tool was shared with six subject experts, and they were requested to assess and rate the tool. Agreement between the experts expressed as CVI and Fleiss kappa statistics in assessing the tool as “Relevant” was calculated and reported.
Agreement proportion measured by averaging method showed 92% agreement and when measured by universal agreement method, it showed 78% agreement. These results support the excellent content validity of the developed MGRAT. However, the CVI of the instrument using universal agreement approach was found to be low compared to the averaging method. This is because of the fact that high number of content experts makes consensus difficult. The developed tool MGRAT is a pioneer attempt to develop a RA tool to assess health risks associated with mass gathering events in Indian settings. We could not find previously reported measures of agreement to compare with the agreement values of MGRAT found in this study. However, content validity studies reported that scale with excellent content validity should be composed of I-CVIs of 0.78 or higher value and S-CVI/UA and S-CVI/Ave of 0.7 and 0.9 or higher value, respectively.,,,
Although content validity index measures are widely used to estimate content validity by researchers, the problems with those measures are that they overlook the percentage of agreement between the experts owing to chance. Therefore, Wynd et al., propose that both CVI and multi-rater kappa statistic should be calculated in the content validity studies because unlike the CVI, kappa adjusts for chance agreement. The chance agreement is an issue of concern while studying agreement indices among assessors, especially when we place four-point scoring within two “relevant” and “not relevant” classes. In other words, kappa statistic is a consensus index of inter-rater agreement that adjusts for chance agreement and is an important supplement to CVI because Kappa provides information about the degree of agreement beyond chance. Nevertheless, the CVI is mostly used by researchers because it is simple for calculation, easy to understand, and provide information about each item, which can be used for modification or deletion of instrument items.,
In the present study, Fleiss kappa statistics to measure the agreement between multiple experts after adjusting the component of chance agreement showed a significant agreement value of 0.522 (95% CI: 0.417, 0.628, P value: 0.001). As per the interpretation of Fleiss' kappa (κ) (from Landis and Koch 1977), 0.522 indicates moderate agreement. Overall, CVI and Fleiss' kappa (κ) both the measures suggest that the content validity of the newly developed tool is appropriate. MGRAT was judged to be valid, appropriate, and feasible to assess the health risks associated with the MG events.
All seven domains were retained after the content validity process. Out of 23 items, 22 items were retained and 1 item was removed after the content validity process. The item “Solid waste management” was merged with “Sanitation and Hygiene facilities” after the content validity process as it was agreed upon as relevant by only one rater and irrelevant by the other five experts. Thus, in the refined MGRAT after content validity, there were 7 domains and 22 items. This validated MGRAT was developed further into a mobile web APP (web-based application) and field tested for its feasibility assessment in one of the religious mass gathering events in Tamil Nadu, India.
| Limitations|| |
Limitations of the study are that the experts' feedback is subjective; thus, the study is subjected to bias that may exist among the experts. If content domain is not well-identified, this type of study does not necessarily identify content that might have been omitted from the instrument owing to the assessment. However, experts are asked to suggest other items for the instrument, which might help minimize this limitation.
This content validity study included a multidisciplinary team of experts. An effort was made to quantify the agreement and efforts to report kappa statistics to assess agreement beyond the chance agreement are some of the advantages of this study. Added to that, the idea of the development of systematic RA tool to assess health risks associated with mass gathering events is itself pioneer effort in India.
Content validity is a systematic process, which includes the judgment/quantification on instrument items by content experts. Such a process should be the leading study in the process of making an instrument to guarantee instrument reliability and prepare a valid instrument in terms of content for the preliminary test phase. To conclude, the present study indicated that the MGRAT is a valid tool, which enjoys an appropriate level of content validity. As the number of rater's increases, there will be difficulty in achieving consensus among all the items, which is the reason for lower CVI/UA when compared with CVI/Ave. Fleiss kappa statistics also indicated moderate agreement among the raters beyond the chance agreement, which also support the appropriate content validity of MGRAT.
We thank Dr. Manoj Murhekar (Director, ICMR-NIE, Chennai), Dr. Sanjay M Mehendale (Ex-ADG, ICMR, New Delhi) and Dr. R Ramakrisnan (Ex-Scientist -G, ICMR-NIE, Chennai) for their support.
Financial support and sponsorship
Upasana Sharma is supported by Indian Council of Medical Research (ICMR) Senior Research Fellowship.
Conflicts of interest
There are no conflicts of interest.
| References|| |
World Health Organization. Public health for mass gatherings: Key considerations. 2015.
Tam JS, Barbeschi M, Shapovalova N, Briand S, Memish ZA, Kieny M-P. Research agenda for mass gatherings: A call to action. Lancet Infect Dis 2012;12:231-9.
Memish ZA, Al-Rabeeah AA. Public health management of mass gatherings: The Saudi Arabian experience with MERS-CoV. Bull World Health Organ 2013;91:899-899A.
Arbon P. The development of conceptual models for mass-gathering health. Prehosp Disaster Med 2004;19:208-12.
Fleischauer AT, Gaines J. Enhancing surveillance for mass gatherings: The role of syndromic surveillance. Public Health Rep 2017;132 (1 Suppl):95S-8S.
Illiyas FT, Mani SK, Pradeepkumar AP, Mohan K. Human stampedes during religious festivals: A comparative review of mass gathering emergencies in India. Int J Disaster Risk Reduct 2013;5:10-8.
Sanyal S, Madan A. (P2-39) Public health safety for traditional mass gatherings in India: A 10-year analysis. Prehosp Disaster Med 2011;26(S1):s(156).
Greenough PG. The KumbhMela stampede: Disaster preparedness must bridge jurisdictions. BMJ 2013;346:f3254.
Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382-6.
Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res 2003;27:94-104.
Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res 2003;25:508-18.
Shi J, Mo X, Sun Z. [Content validity index in scale development]. Zhong Nan Da Xue Xue Bao Yi Xue Ban 2012;37:152-5.
Polit DF, Beck CT. The content validity index: Are you sure you know what's being reported? Critique and recommendations. Res Nurs Health 2006;29:489-97.
Blais MA, Hilsenroth MJ, Castlebury FD. Content validity of the dsm-iv borderline and narcissistic personality disorder criteria sets. Compr Psychiatry 1997;38:31-7.
Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar A-R. Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. J Caring Sci 2015;4:165-78.
[Table 1], [Table 2]