Document is current

Current Referee Status

Referee status: Indexed

Article Reports

Referee Report: https://doi.org/10.21956/wellcomeopenres.14575.r31073, Scott D. Nash, Trachoma Control Program, Carter Center, Atlanta, GA, USA, 12 Mar 2018, version 1, indexed

Referee Comment: <b>Robert Butcher</b>; <i>Posted: 13 Jul 2018</i>; We appreciate the time Dr Nash has taken to appraise this manuscript. We include a point-by-point response to his comments below. We believe his feedback has improved the article, and hope he agrees. &nbsp; Graders were trained according to the GTMP training protocol as described elsewhere. The training was robust; trainees attended a taught course on trachoma, were required to achieve a kappa score of ≥0.7 when grading a set of 50 photographs in which the trachoma grade had been agreed by several experts and were required to achieve a kappa score of ≥0.7 against a highly experienced grader trainer in the assessment of 50 children aged 1–9 years in the field. Both graders working on this study had met those criteria on the first attempt during a GTMP training in the Solomon Islands in September 2013 (unfortunately we do not have their specific kappa scores available). The grading in Temotu (where 11 of the 13 study villages were located) was undertaken by a single grader who had also successfully completed a GTMP training course in Ethiopia prior to the Solomon Islands course. A second grader conducted the grading in Rennell &amp; Bellona (2 of the 13 study villages were in this province). We believe by training in this manner, we have ensured our data is as comparable as possible to data generated by the GTMP, and by impact and pre-surveillance surveys in the majority of trachoma-endemic areas. Data generated by graders trained in this manner are used to guide treatment decisions around the world. However, to support our research programme alongside the Solomon Islands National Trachoma Control programme we have collected photographs to enable us to retrospectively check grading. In a sample of photographs from a previous survey, the kappa score between field grade of TF and photograde of TF was 0.88. In this survey the photographs were graded with the FPC system rather than the simplified system. The kappa agreement between a photograph grade of F2/F3 and field grade of TF was 0.52, indicating moderate agreement. This is lower than in previous studies in the Solomon Islands, potentially because of more borderline cases in a post-treatment community or because F2/F3 is not exactly comparable to TF. We have expanded the methods section in version 2 to give a fuller account of the training process, we have added the F2/F3 vs TF kappa score to the results, and have added a section in the discussion regarding this process. &nbsp; Two graders were involved in this study. Both had met the predefined criteria for successful grading and were therefore accredited to grade trachoma to the quality required to guide treatment decisions. The kappa agreement between field and photograph grades along with caveats about expectations for that agreement has been included in the manuscript in line with the previous comment. The most common type of disagreement was field+ / photograph-, therefore there is a possibility that that grader had overcalled some milder cases of follicular inflammation as TF. For the reasons described above, we expect a certain number of discrepancies. Also, when a discrepancy occurs, we believe the field grade to be more likely to be accurate as the field grader can achieve a three-dimensional real-time view of the eyelid. It is worth noting that the key finding of this paper that the majority of TF cases do not have antibodies to Pgp3 holds true if we define the clinical phenotype using the photograph grades (70% of children whose photographs were graded as F2/3 were Pgp3 seronegative, chi-squared test of association p = 0.49). &nbsp; It wasn't possible to recheck grading in the field, however, we have presented the comparisons between field and photograph grade above and in the paper. &nbsp; Because specific interventions at the programmes are guided by TF, TI and TT grades but not TS grades, scale up of mapping activities has focussed on those signs. Therefore, grading for TF, TI and TT has been repeatedly replicated in thousands of districts round the world, whereas TS grading is relatively uncommon. Therefore, while graders may be able to be trained to assess TS reliably, training for TF, TI and TT is ‘standardised’. We have amended the terminology in the methods to reflect this, and added a reference where this is also discussed. &nbsp;

Referee Report: https://doi.org/10.21956/wellcomeopenres.14575.r31070, Meraf A. Wolle, Dana Center for Preventive Ophthalmology, Wilmer Eye Institute, Johns Hopkins University School of Medicine, Baltimore, MD, USA, 14 Mar 2018, version 1, indexed

Referee Comment: <b>Robert Butcher</b>; <i>Posted: 13 Jul 2018</i>; We are grateful to Dr Wolle for her feedback on our article. We have taken her comments into account as we prepared the second version of this manuscript, and believe they have enhanced the quality and transparency of the article. We hope the following additional information addresses some of her comments directly. The photograph graders had to achieve a kappa score of &gt;0.7 in C grading, a kappa score of &gt;0.7 in F grading and a kappa score of &gt;0.7 in P scoring when compared to a highly experienced grader before they were considered to be ‘validated’ for photograding. It is hard to calculate a kappa score between graders in this particular manuscript because those where the primary graders did not reach consensus were arbitrated. Where the first two grades agreed, the grade was accepted as the final grade. Where the first two grades disagreed on F, P or C, an independent expert adjudicated. There are, therefore, a subset of photographs where both primary graders have called the same C grade, and the arbitrator has also graded C. In these cases, the kappa agreement was 0.76 between primary and arbitrator grades. 42% of participant photographs did not agree on F or P or C score, so were arbitrated. The majority of these were due to disagreement in P score. We have presented data in the article from a comparable study in Guinea Bissau where the same grading process was used, and a starkly different picture emerges. In addition, we have listed below some examples of recent studies using the FPC system. Ramadhani <i>et al.</i> Immunofibrogenic Gene Expression Patterns in Tanzanian Children with Ocular Chlamydia trachomatis Infection, Active Trachoma and Scarring: Baseline Results of a 4-Year Longitudinal Study. <i>Front. Cell. Infect. Microbiol.</i>, 2017 Zhou <i>et al. </i>The conjunctival microbiome in health and trachomatous disease: A case control study. <i>Genome Medicine</i>, 2014. Roberts <i>et al.</i>, Conjunctival Scarring in Trachoma Is Associated with the HLA-C Ligand of KIR and Is Exacerbated by Heterozygosity at KIR2DL2/KIR2DL3. <i>PLoS NTDs</i>, 2014. While we are aware of other grading systems developed specifically for grading scarring in photographs, we considered this grading system to be the best balance between the relatively simplistic tool of the simplified system and the highly complex tools described by others. We felt the experience our group had using the FPC system would lead to more reproducible grading which would outweigh the benefit incurred by using more specialised methods. We acknowledge that the difference between borderline cases of C0 and C1 or between C1 and C2 may be difficult to arbitrate, although felt that was a problem common to all trachoma grading, for example with regards to interpretation of what constitutes a follicle &gt;0.5mm or the difference between different levels of inflammation. Regarding the question of distortion of the tarsus, we believe in the photographs where C3 was called (examples of which are displayed in figure 1) the distortion of the tarsus is clear, however, it is possible that some cases of C3 were missed. We believe at the population level, this is unlikely to lead to gross shifts in results, especially in the context of this population where over 85% of people show no scarring at all. Finally, the comparator data we present in figure 2b demonstrate that the same grading scheme used in an area with hyperendemic infection does identify cases of moderate and severe scarring. In the comparator population from Guinea Bissau, the photographs were reviewed by two experienced graders who met to discuss the outcome of discrepant results. Further information about this process was not recorded, so it is not possible to determine what proportion needed to be discussed. The kappa agreement score between photograde of F2–3 and field grade of TF is 0.52 and additional information surrounding this comparison is included in response to Dr Nash’s comment. The two hypotheses that have emerged as a result of this set of work are (1) that an as-yet uncharacterised, non-bacterial cause of follicular conjunctivitis is inflating estimates of TF or (2) that the host response of residents of the Solomon Islands to circulating <i>Ct </i>infection is not the same as that in other countries, so does not result in the antibody and cicatrizing responses seen in other endemic areas. We consider the former to be more likely, and mention that in the final sentence of the second paragraph in the discussion. Thank you for directing us to the paper from West <i>et al.</i>. Our interpretation of the data in that paper is that the small proportion of people classed as sero-reverters at one year actually have a distinguishably lower antibody level at baseline than those who were positive at both timepoints, and a distinguishably higher antibody level at follow up than those classed as negative at both timepoints, therefore we believe there is no evidence for ‘complete’ seroreversion, and the statement should stand.

Grant Information: This work was supported by the Wellcome Trust grant [098481] awarded to MJB, an Intermediate Clinical Fellowship [098521] awarded to AWS, supporting AWS and RB, and an Institutional Strategic Support Fund [105609] awarded to ChR. This work was also supported by the Fred Hollows Foundation, Australia [1041] funding the field and laboratory costs. OS, KJ, LS and CR were employed by the Solomon Islands Ministry of Health and Medical Services for the duration of the survey. DLM receives funding through the US Agency for International Development through an interagency agreement with CDC. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.