|Year : 2013 | Volume
| Issue : 2 | Page : 173-177
Assessment of validity and reliability of IMNCI algorithm in comparison to provisional diagnosis of senior pediatricians in a tertiary hospital of Kolkata
Agnihotri Bhattacharyya1, Shuvankar Mukherjee1, Chitra Chatterjee2, Samir Dasgupta3
1 Department of Community Medicine, Calcutta National Medical College, Kolkata, India
2 Department of Community Medicine, College of Medicine and Jawaharlal Nehru Memorial Medical Hospital, Kalyani, WBUHS, Kolkata, India
3 Department of Community Medicine, Medical College, Kolkata, India
|Date of Web Publication||31-Aug-2013|
Tobin Road, Rajpur, P.O. Rajpur, 24 PGS(S), West Bengal - 700 149
Source of Support: None, Conflict of Interest: None
Background: Integrated management of childhood illness (IMNCI) is already operational in many states of India, but there are only limited studies in Indian scenario comparing its validity and reliability with the decisions of pediatricians. Aims and Objectives: To assess the validity and reliability of the IMNCI algorithm with provisional diagnosis of senior pediatricians for each IMNCI classifications. Materials and Methods: The present study is done with all the young infants between 0-2 months presented during the study period with a fresh episode of illness to test the validity and reliability of the algorithm in comparison to provisional diagnoses of senior pediatricians. The study was done in a tertiary care hospital. Validity characteristics such as sensitivity, specificity, positive predictive value, negative predictive value, and reliability characteristics such as percent agreement and Kappa were assessed for individual IMNCI classifications. Results: The sensitivity of possible serious bacterial infection, local bacterial infection, jaundice, no dehydration and possible serious bacterial infection, not able to feed were 88.89, 14.29, 66.67, 25 and 44.44% respectively. The specificities for the same conditions were 71.72, 99.09, 99.07, 94.50 and 86.87%. Percent agreements for similar conditions were 74, 94, 97, 90 and 80% respectively and the Kappa ratios were 0.38, 0.20, 0.73, 0.19 and 0.29 respectively. Conclusion: It could be concluded that IMNCI is quite a sensitive strategy and could identify severe illnesses of young infants requiring referral to higher facility. Further studies, particularly in primary health care setting, are required.
Keywords: IMNCI algorithm validation, Kappa, young infants
|How to cite this article:|
Bhattacharyya A, Mukherjee S, Chatterjee C, Dasgupta S. Assessment of validity and reliability of IMNCI algorithm in comparison to provisional diagnosis of senior pediatricians in a tertiary hospital of Kolkata. J Family Med Prim Care 2013;2:173-7
|How to cite this URL:|
Bhattacharyya A, Mukherjee S, Chatterjee C, Dasgupta S. Assessment of validity and reliability of IMNCI algorithm in comparison to provisional diagnosis of senior pediatricians in a tertiary hospital of Kolkata. J Family Med Prim Care [serial online] 2013 [cited 2019 Aug 18];2:173-7. Available from: http://www.jfmpc.com/text.asp?2013/2/2/173/117393
| Introduction|| |
Children brought for medical treatment are often found to be suffering from more than one morbid condition, making a single diagnosis impossible. The effective management of those conditions is more dependent on adopting a holistic approach using cheap, universally available and accessible strategies rather than sophisticated and expensive technology. According to the World Bank Report 1993, for situations where laboratory support and clinical resources are limited, such an approach is more realistic and cost-effective, and therefore, has the potential to make the greatest impact on the global burden of disease. 
During the year 1992, the World Health Organization (WHO), WHO, in collaboration with United Nations Integrated Children's Emergency Fund (UNICEF), and some other agencies, institutions and individuals, responded to the challenge by adopting a strategy known as integrated management of childhood illness (IMCI).  IMCI is an evidence-based syndromic approach. In this approach, priority has not been given to diagnosis of individual disease, rather the classification of the diseases and assessment of severity according to the common signs and symptoms were approached. Along with curative care for common childhood illnesses like acute respiratory infection, diarrhea, measles, malaria and malnutrition, the strategy also addressed aspects of nutrition, immunization, and other important elements of disease prevention and health promotion.  IMCI addressed the age group between 2 weeks and 59 months, but in India it was found that neonatal mortality constitutes 64% of "under-five mortality", all neonates are included in the strategy, starting from the day of birth and it is adapted in the Indian version as IMNCI. In the adapted version, the entire age group of 0 to 59 months (as against 2 weeks to 59 months in IMCI) was included to address the neonatal mortality challenge.  In the IMNCI version, the age group was divided in two groups as 0-2 months and 2 months to 59 months. In both the age groups, through the combination of signs and symptoms classification of the condition is made. This classification indicates the severity of conditions and call for specific actions like red indicating urgent referral, yellow indicating management at existing health facility and green indicating management at home.
The IMCI algorithm in both 7 days to 2 months and 2 months to 5 years age group was validated in India and other developing countries by several studies. ,,
As IMNCI is the adapted version of IMCI, followed only in India, there is paucity of published study testing the validity of IMNCI. However, two studies were done at Department of Pediatrics, Kalawati Saran Children's Hospital and Lady Hardinge Medical College testing the validity of IMNCI algorithm during 2002-2003 by Goswami V, Singh V, Dutta AK  and by Kaur S, Singh V, Dutta AK, Chandra J during the year 2005-2006. 
In West Bengal IMNCI algorithm was operational in Purulia district since the year 2008, but there is hardly any published study testing its validity. Under this background, the present study has been done with the objective to assess the validity and reliability of the algorithm with provisional diagnosis of senior pediatricians for each IMNCI classifications.
| Materials and Methods|| |
This observational, cross sectional study was done at department of Pediatrics, in a tertiary care hospital of Kolkata during January to March 2009 with the young infants between 0-2 months. All the young infants presented during this study period with a fresh episode of illness were included in the study with o an informed consent from their parents. Altogether 117 mothers were interviewed with a pre tested, semi structured interview schedule and the young infants were assessed, classified and categorized for treatment using Physician's chart booklet for IMNCI along with the assessment form used for IMNCI. These cases were then sent to the pediatricians for further assessment without any mention of the classifications made by the researchers using the IMNCI algorithm so as to reduce observer bias. Pediatricians examined the young infants as they did in any tertiary care hospital and recorded the presenting symptoms, clinical features and provisional diagnosis in the OPD or emergency examination tickets. The provisional diagnosis was based only on clinical evaluation. In the present study, the IMNCI classifications were compared with those provisional diagnoses. The provisional diagnosis which could be compared with a particular IMNCI classification was determined after discussion with senior faculty members of department of Pediatrics, Medical College, Kolkata.
Validity characteristics like sensitivity, specificity, positive predictive value, negative predictive value, and reliability characteristics like percent agreement, and Kappa were assessed for individual IMNCI classifications against Pediatrician's provisional diagnosis. 
The Kappa test is used to exclude the extent of percent agreement which was due to chance. The result of Kappa has some arbitrary division like.
Under 0.2 is negligible agreement.
0.2 to 0.4 is minimal agreement.
0.4 to 0.6 is fair agreement.
0.6 to 0.8 is good agreement.
And over 0.8 is excellent agreement 
| Results|| |
Out of the 117 young infants, 25.64% were within 7 days of age and 2.56% were presented within 24 hours. The young infants were presented with one or multiple presenting symptoms and among them cough and cold (33.33%) was the main symptom. Next were fever (12.82%), loose stool (11.11%) and respiratory distress (11.11%). Other important presenting symptoms were convulsion (7.69%), vomiting (7.69%), yellow discoloration of skin (5.98%), inability to suck (5.98%), unsatisfactory feeding (5.13%), and no cry after birth (2.56%).
All the young infants were assessed and classified according to the IMNCI algorithm. The common classifications were severe malnutrition (23.08%), feeding problem (22.22%), low weight for age (21.37%), possible serious bacterial infection (19.66%), possible serious bacterial infection, not able to feed (18.80%), severe jaundice (1.71%) etc [Table 1].
|Table 1: Distribution of study subjects according to the IMNCI classification (multiple classification) n = 117 |
Click here to view
When pediatricians assessed the cases, a single provisional diagnosis was made for each study subject based on initial clinical evaluation.
The important provisional diagnoses as found in the OPD or emergency tickets were common cold (22.20%), followed by septicemia (15.40%), jaundice (7.70%), breast fed loose stool (6.80%), birth asphyxia (6.80%), upper respiratory tract infection (6.80%), local infection (6%), lower respiratory tract infection (4.30%), low birth weight baby (3.50%), thrush (1.70%) [Table 2].
|Table 2: Distribution of study subjects according to provisional diagnosis by pediatricians n = 117 |
Click here to view
In eight cases (6.80%), the pediatricians found no abnormality and the initial diagnoses were healthy baby. For fourteen cases (12%), there were some other single provisional diagnoses like spina bifida occulta, anorectal malformation, cephalhematoma, congenital hypothyroidism, Down's phenotype, abdominal colic, congenital leukemia, stenosis of gut, hemorrhagic disease of newborn, meconium aspiration syndrome, excessive jitteriness, prune belly syndrome, congenital cyanotic heart disease [Table 2].
The classifications made by IMNCI algorithm were compared with the provisional diagnoses of the pediatricians. As per the opinions of senior faculty members of pediatrics department, possible serious bacterial infection was comparable with septicemia; local bacterial infection with cellulites, boil; whereas for jaundice, low body temperature, and diarrhea, the comparisons should be with similar conditions. There was no diagnosis as severe jaundice, severe dehydration, severe malnutrition, low weight for age, feeding problem and no feeding problem by the pediatricians and so comparison could not be made for those conditions. Both IMNCI algorithm and pediatrician's provisional diagnosis did not find any case of some dehydration, severe persistent diarrhea or severe dysentery and obviously no comparison could be done for those cases.
In case of possible serious bacterial infection, in the present study, sensitivity was found as 88.89%, positive predictive value was 36.39%, negative predictive value was 97.26% and specificity was 71.72% [Table 3]. Along with validity, the reliability of the classification was also assessed by percent agreement, by exclusion of chance by Kappa test.  The percent agreement was 74%. But it was evident that most of the agreement was due to chance; so when chance agreement was excluded by Kappa test, the Kappa value was only 0.38 [Table 3], which indicates only minimal agreement. 
In case of local bacterial infection, sensitivity was only 14.29% whereas the specificity was 99.09% [Table 3]. The positive and negative predictive value was 50 and 94.78% respectively. The percent agreement of local bacterial infection was 94% with the Kappa value 0.20, indicating only minimal agreement  [Table 3].
|Table 3: Distribution of validity and reliability characteristics of IMNCI classification, against pediatrician's provisional diagnosis n = 117 |
Click here to view
In cases of jaundice, sensitivity was 66.67% whereas the specificity was 99.07% [Table 3]. The positive and negative predictive value was 85.71 and 97.27% respectively. The percent agreement was 97% with Kappa value 0.73, indicating good agreement  [Table 3].
For no dehydration classification in algorithm, the sensitivity was 25% whereas the specificity was 94.50% [Table 3]. The positive and negative predictive value was 25 and 94.50% respectively. The percent agreement was 90%, the Kappa value was only 0.19, which indicates only negligible agreement  [Table 3].
As per the algorithm, there were 44 cases of possible serious bacterial infection and among those, there were 21 cases, which had no ability to feed or suck. Though in the algorithm, those cases were classified separately, pediatricians did not consider them as separate entity and diagnosed as septicemia as a whole. This might be the reason for this low sensitivity as 44.44%. The specificity in those conditions was 86.87% [Table 3]. The percent agreement was 80%. However, when chance agreement was excluded by Kappa test, the Kappa value was only 0.29, indicating only minimal agreement  [Table 3].
| Discussion|| |
Integrated management of neonatal and childhood illness (IMNCI) is already operational at the field level in India, but there is paucity of published study testing its validity and reliability.
In the present study, it was found that young infants were presented with one or multiple symptoms and among them cough and cold (33.33%) was the main symptom. The other important presenting symptoms were fever (12.82%), loose stool (11.11%) and respiratory distress (11.11%). In the study done by Sachdev et al. the most common presenting symptom was cough (65.3%), followed by fever (53.8%), running nose (40.3%), diarrhea (17.3%), respiratory distress (9.6%).
The young infants were assessed and classified according to the IMNCI algorithm and it was found that majority (54.70%) had single classification, 38.47% had two classifications, 5.98% had three and 0.85% had four classifications. The mean number of classifications was 1.53 with standard deviation 0.65. In the study done by Sachdev et al, the mean number of classification by IMNCI algorithm was 1.8 with standard deviation 0.8. In the study done by Kaur, Singh, Dutta, Chandra,  the mean number of morbidities was 1.75.
In the present study, the important provisional diagnoses as found in the OPD or emergency tickets were common cold (22.20%), followed by septicemia (15.40%). In the study done by Sachdev et al. the most common diagnosis of the pediatricians was low birth weight (75.20%), followed by diarrhea (27.9%), upper respiratory tract infection (26.3%) and septicemia (21.7%).
In case of possible serious bacterial infection, in the present study, sensitivity was found as 88.89%, positive predictive value was 36.39% and specificity was 71.72%, whereas in the study done by Sachdev et al, the sensitivity was 96.5% and specificity was 51.8%. In the study done by Kaur, Singh, Dutta, Chandra,  the sensitivity of algorithm to identify bacterial infection was 88.5% while the specificity was relatively low (57.4%).
The reasons for this low specificity and positive predictive value might be that the predictors for possible serious bacterial infection as mentioned in the algorithm could predict other conditions also as convulsion or bulging fontanelle might occur in other CNS disorders like hypoxic-ischemic encephalopathy, birth asphyxia, intracranial hemorrhage, meningitis, hypoglycemia, hypocalcaemia or any other causes of subdural effusion apart from septicemia.  Similarly increased respiratory rate, severe chest indrawing or nasal flaring could be the manifestation of hyaline membrane disease, transient tachypnea of newborn, meconium aspiration syndrome and other causes of respiratory insufficiency.
Dehydration fever, which is very much common particularly in an overheated nursery, might be an important cause of over diagnosis of possible serious bacterial infection. In the study by Goswami, Singh, Dutta,  algorithm tends to over diagnose serious bacterial infection by 8-20% (in three age groups). 
In the study done by Kaur, Singh, Dutta, Chandra,  also, it was found that out of 80 cases classified by the algorithm as possible serious bacterial infection with difference in diagnosis with the pediatricians, 31 (38.7%) had birth asphyxia with hypoxic-ischaemic encephalopathy, 16 (20%) had hypocalcemic seizures, 11 (13.7%) had meconium aspiration syndrome, and 7 (8.8%) had hemorrhagic disease of newborn. Other conditions included respiratory distress syndrome (9 cases), transient tachypnea of newborn (4 cases), and neonatal seizures (2 cases). 
Pediatricians diagnosed one big boil and more than 10 pustules as boil or cellulites, comparable with local bacterial infection whereas for the IMNCI algorithm, it was considered as possible serious bacterial infection. This difference in detection might be the cause of low sensitivity.
In cases of jaundice, sensitivity was 66.67% whereas the specificity was 99.07. Pediatricians diagnosed all relevant cases as jaundice; clinically they did not categorize any case as severe jaundice. However, in the algorithm when the jaundice appears in the 1 st day of life or persists for more than 14 days and extends in the palms and soles, it was classified as severe jaundice.  In the study done by Kaur, Singh, Dutta, Chandra,  the algorithm under-diagnosed the severity of jaundice in few subjects (12/131) and over-diagnosed (8/131) the severity in few subjects.
For no dehydration classification in algorithm, the sensitivity was 25% whereas the specificity was 94.50%. Probability of low sensitivity might be due to some cases like lactose intolerance, breast-fed loose stool and not consideration of urination status by the algorithm. In the study done by Kaur, Singh, Dutta, Chandra,  of the 76 cases identified as diarrhea by the algorithm, 22 (29%) had breast-fed stools.
In the algorithm, some conditions like birth asphyxia, Down's phenotype, abdominal colic, stenosis of gut, hemorrhagic disease of newborn, meconium aspiration syndrome, excessive jitteriness, prune belly syndrome, congenital cyanotic heart disease, anorectal malformation, cephalhematoma, congenital leukemia, breast-fed loose stool or lactose intolerance, spina bifida occulta, and congenital hypothyroidism were not covered. In the study done by Kaur, Singh, Dutta, Chandra,  also it was seen that the algorithm under diagnosed some surgical conditions and congenital anomalies. 
Therefore, in conclusion, it could be mentioned that IMNCI is a quite sensitive strategy and could identify the severe illnesses of the young infants requiring referral to higher facility. Presence of other diagnosis with similar symptoms might result in false positive errors and low specificity. The algorithm covered most of the conditions, except some uncommon and rare ones. However, as this study was done in a tertiary care setting, further study particularly in primary health care setting is required.
| References|| |
|1.||World Bank. World development report 1993: Investing in Health. New York: Oxford University Press; 1993. |
|2.||IMNCI Training Module (No.1) for MOs. New Delhi, UNICEF, 2005. |
|3.||Child Health Programme in India - Major milestones in Child Health [Internet]. Ministry of Health and Family Welfare, Government of India. Available from: http://www.mohfw.nic.in/dofwwebsite/childhealth rti.pdf [Last cited on 2009 Jun 03]. |
|4.||Perkins BA, Zucker JR, Otieno J, Jafari HS, Paxton L, Redd SC, et al. Evaluation of an algorithm for integrated management of childhood illness in an area of Kenya with high malaria transmission. Bull World Health Organ 1997;75(Suppl 1):33-42. |
|5.||Kalter HD, Schillinger JA, Hossain M, Burnham G, Saha S, de Wit V, et al. Identifying sick children requiring referral to hospital in Bangladesh. Bull World Health Organ 1997;75(Suppl 1):65-75. |
|6.||Gupta R, Sachdev HP, Shah D. Evaluation of the WHO/UNICEF Algorithm for Integrated management of childhood illness between the age of one week to two months. Indian Pediatr 2000;37:383-90. |
|7.||Goswami V, Singh V, Dutta AK, Chandra J. Evaluation of simple clinical signs of illness in young infants (0-2 months) and its correlation with WHO IMCI algorithm (7 days-2 months). Indian Pediatr 2006;43:1042-9. |
|8.||Kaur S, Singh V, Dutta AK, Chandra J. Validation of IMNCI algorithm for young infants (0-2 months) in India. Indian Pediatr 2011;48:955-60. |
|9.||Jeckel JE, Katz DL, Elmore JG. Understanding errors in clinical medicine. In: Epidemiology, biostatistics, and preventive medicine. 2 nd ed. Philadelphia: W. B. Saunders Company; 2001. p. 108-14. |
|10.||Stoll BJ, Kliegman RM. Fetus and the neonatal infant. In: Nelson textbook of pediatrics. Philadelphia: WB. Saunders; 16 th ed. 1996. p. 487-545. |
|11.||IMNCI Training Modules (No. 2) for MOs, UNICEF, New Delhi, 2005. |
[Table 1], [Table 2], [Table 3]