Document is current

Current Referee Status

Referee status: Indexed

Article Reports

Referee Report: https://doi.org/10.21956/wellcomeopenres.19477.r48096, Glen Dario Rodriguez Rafael, Unidad de Postgrado de la Facultad de Ingenieria de Informàtica y Sistemas, Universidad Nacional Mayor de San Marcos, Lima, 21 Jan 2022, version 4, 1 approved, 1 approved with reservations

Referee Report: https://doi.org/10.21956/wellcomeopenres.19477.r48856, Ayoub Bagheri, Utrecht University, Utrecht, 14 Oct 2022, version 4, 1 approved, 1 approved with reservations

Referee Comment: <b>Rodrigo M Carrillo-Larco</b>; <i>Posted: 16 Oct 2022</i>; We are thankful for the insightful comments by the reviewer. We appreciate the time he invested in reading our work. Please see our point-by-point answers below. We hope these satisfy the reviewer’s expectations. <b>C1. Please provide a link to your toolkit as it is mentioned that it is available to use for other researchers.</b> R1. We have uploaded the Jupyter Notebooks for these analyses to the Figshare repository reported in the Data Availability statement, which was updated accordingly. The sentence in the introduction about the ‘toolkit’ was removed. <b>C2. Could you elaborate on the contributions of this work?</b> R2. Yes, the contribution of this work has been further elaborated in the Introduction and Discussion sections. The new lines read: [introduction] <i>Our work contributes to the NLP health literature by leveraging on standard and well-known algorithms to analyse health policy/politics documents. Because most previous NLP health research focused on electronic health records (i.e., clinical oriented) or social media (i.e., population health oriented), our work expanded the use of NLP in health research to new documents (policy/politics documents) with potential public health implications. Furthermore, another contribution of our work also lies in the fact that we used a sort of text documents that, arguably, may be available to a much broader audience. Even though electronic health records and social media are valuable sources of information for NLP analyses, electronic health records may be seldom available in low- and middle-income countries where people could also engage less in social medial. Nevertheless, in most countries during election period there may be some sort of documents summarizing the proposals by the candidates who may share these as much as possible for electors to familiarize themselves with the candidates.</i> [discussion] <i>Finally, the contribution of this work lies in the fact that we expanded the data sources frequently used in NLP health research, which has mostly leveraged on texts from electronic health records and social media. Unfortunately, electronic health records may not be available in many low- and middle-income countries, where people may not engage as much in some social media outlets as their peers in high-income countries. Thus, our work may democratize the use of NLP in health research by leveraging on a new data source (e.g., materials distributed during national elections) that, arguably, is accessible to a much broader population. Future work, moving from electronic health records and social media, could focus on texts from newspapers and open-access documents published by authorities such as reports from government officers, summaries from government meetings, and summaries of debates at the congress; again, all these sources may be available to a large population above and beyond electronic health records and social media.</i> &nbsp; <b>C3. I would like to know why the authors choose these methods and how do they compare them with each other.</b> R3. We have provided this information in the Methods section, right before the description of the algorithms. The new lines read: <i>We used five NLP algorithms: TF-IDF, LDA, TextRank, KeyBERT, and Rake. We chose these algorithms because they are frequently used in the NLP literature, they are available through most libraries in Python and R facilitating the replication and outreach of our work, and they satisfied our goals (i.e., to describe the health chapters in government plans </i>by summarising words and terms <i>). </i> <i>TF-IDF and TextRank reveals keywords; TF-IDF does so based on the relative frequency of each term whereas TextRank selects keywords based on a trained model. Compared to TF-IDF and TextRank, KeyBERT as well as Rake reveal keywords or short phrases (e.g., 3-5 words) based on trained models. LDA, on the other hand, groups texts into unknown clusters and the user labels such clusters (i.e., as in unsupervised machine learning with K-means). Altogether, these algorithms provide a framework to describe the health chapters in government plans going from unique terms (TF-IDF and TextRank), to short phrases (KeyBERT and Rake), and grouping into unknown clusters (LDA).</i> <b>C4. Is it an improvement if you are using a lemmatisation method in TF-IDF and LDA?</b> R4. We followed the standard recommendations for TF-IDF and LDA as shown in references 25, 26, and 27 in the manuscript. For both algorithms, lemmatisation was suggested before implementing them. <b>C5. What is the reasoning when you cluster data into only two groups in LDA? It would be nice if you look at 5, 10, 20, and 50 as well.</b> R5. We have justified our choice of two (02) groups in LDA. The new lines read (new text underlined): <i>For this work, we defined two groups (i.e., we specified that the LDA algorithm should create two groups following an unsupervised approach from the data). <u>Two is the absolute minimum number of groups and we chose such option this being an exploratory work compounded with other NLP algorithms, and we did not focus solely on the LDA results. For a discussion on this limitation, please refer to the Strengths and Limitations section.</u></i> We also discussed this decision, specifically in Strengths and Limitations section: <i>Sixth, the LDA algorithm was applied with the absolute minimum number of groups (i.e., to identify two clusters). We could have tried to find 5, 10, 20 or more groups; however, dividing the texts in many thin groups would have not been ideal. First, we only worked with the health chapters which were, on average, one or two pages long; in other words, these chapters did not have much content to be dissected into many clusters. Had we analysed the full plans (not only the health chapters), finding more groups would have helped to separate all sections. Second, the government plans were concise straight-to-the-point documents (sometime bullet points); that is, they did not cover several topics nor in detail. This made it difficult to dissect the texts into many thin groups.</i> <b>C6. Could you explain in the paper what is the input to the LDA model?</b> R6. We added these lines: <i>The input for training the LDA algorithm was all the government plans (separately for each year), having removed the stop words and lemmatized the terms. Then, with the cleaned (lemmatized and without stop words) text for each political party, the model assigns each party to a group defined during the LDA training. </i> <b>C7. What are the limitations of using KeyBERT? Do you have to use fixed vectors?</b> R7. The question was not very clear to us. We apologise unreservedly and hope this answer addresses the issue. We added these lines in the Methods section where we described the KeybERT analysis: <i>A limitation of this algorithm is that it retrieves a maximum of six keywords; though this did now affect our work because we sought five keywords as it was the case for other algorithms. </i> <b>C8. Could you provide the equation for the rake method and describe what was the library you used for this?</b> R8. We added these lines: <i>We used the class Rake from the rake_nltk library; for further details including the equation for the Rake method please refer to reference 15.</i> <b>C9. If you look at the frequencies, words similar to 'health' should appear in most of the documents, therefore with TF-IDF they should gain a very low weight. How can you respond to this argument? Isn't 'health' a stopword in this analysis?</b> R9. We have included the term ‘health’ as a stop word (i.e., excluded the term ‘health’ from the TF-IDF analysis). The TF-IDF figures and results have been updated accordingly. Notably, the main results did not change. Following this recommendation, and because TF-IDF has the same pre-processing as the LDA algorithm, the term ‘health’ was also excluded from the LDA analysis. We updated the results and table accordingly. The term ‘health’ was not removed from the other analyses because they use the semantic context of the text and removing a word would go against this recommendation. Moreover, removing the word ‘health’ could change some relevant key words or short phrases such as ‘public health’, ‘universal health’, or ‘health insurance’. <b>C10. In Table 2, would you provide the probabilities for each term in groups as well?</b> R10. Table 2 now includes the probabilities for each of the ten terms. Please, keep in mind that the terms have changed as well (not the same as in the original submission), because the LDA analysis was re-ran having removed the word ‘health’ (i.e., ‘health’ is now a stop word). <b>C11. How do you compare Fig. 2b and Fig. 3b?</b> R11. We have included these lines in the Results section, at the end of the LDA sub-heading. <i>Figure 2 shows the LDA results for the government plans in 2016 and Figure 3 for the plans in 2021. While these figures show that political parties are more inclined to either Group 0 or Group 1, a direct comparison between 2016 and 2021 is troublesome because: i) the same exact political parties did not participate in the two elections; ii) the composition of the political parties in terms of roles, interests and priorities may have changed between 2016 and 2021; and iii) amidst the COVID-19 pandemic, the focus of the health-related government plans would be different between 2016 and 2021. Perhaps in response to the COVID-19 pandemic, the difference observed between Figure 2B and Figure 3B could signal that the 2021 plans focused more on terms related to the structure of the health system (i.e., Group 1 appeared to cluster terms that are related with the structure of the health system).</i> <b>C12. The keywords for Fig. 4c are not clear.</b> R12. Unfortunately, we could not customize the very first label for Figure 4C. The very first label was a three-word term which font/size we could not customize without compromising the other labels. Even though we tried different methods to change that label only, we could not find an optimal solution. We apologise for the inconvenience. Happily, the message and interpretation of Figure 4C was not altered. <b>C13. From page 8 until 26 you just see figures and tables with no text explaining them. </b> R13. The text explaining or describing the figures and tables are just before the pages containing only figures and table. This was the layout decided by the journal’s typesetters for the PDF file. However, the HTML version of the manuscript has the figures and tables embedded in the text. We apologise for the inconvenience and we will work with the editors to improve the layout of the PDF version. <b>C14. The first point in the discussion is about applying novel techniques in this study. I must say that I do not agree with the authors as I do not see any novel method in the paper. The paper does not propose any novelty (that, of course, is not the focus of the study and the journal), while except KeyBERT (2020), the other methods are quite used in many studies: TF-IDF (1972), LDA (2003), Rake (2010), textrank (2004).</b> R14. Indeed. We appreciate the discrepancy and the opportunity to clarify. We modified the very first sentence in the Discussion (new text underlined): <i><u>We used standard and well-known NLP algorithms</u></i> <i> to study health chapters of the government plans of the political parties in the 2016 (18 plans) and 2021 (19 plans) general elections in Peru. <u>We did not aim to develop new NLP algorithms nor to modify the available algorithms, rather, we aimed to apply available NLP algorithms in a novel topic: health policy/politics.</u></i> <b>C15. Simply because you are looking at the health chapters, TF-IDF should remove the term 'health' from the analysis, unless you say that many of the documents do not contain the word health.</b> R15. The word ‘health’ was removed from the TF-IDF and LDA analyses (i.e., it was set as a stop word). The results, figures, and tables for the TF-IDF and LDA analyses were updated accordingly.

Referee Report: https://doi.org/10.21956/wellcomeopenres.20483.r113436, Shaina Raza, Vector Institute, Toronto, Ontario, 07 Jan 2025, version 5, indexed

Grant Information: This work was supported by the Wellcome Trust Interna-tional Training Fellowship[214185/Z/18/Z to RMC-L].The funder had no role in this work and decision to submit this work for publication. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.